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San Diego Striving Readers Project 


Year 4 Project Evaluation Report 


I. EXECUTIVE SUMMARY OF FINDINGS: IMPLEMENTATION AND IMPACT 


Since the inception of the Striving Readers grant program in 2006-07, the San Diego 
Unified School District (SDUSD) has implemented an innovative adolescent literacy 
program, called Strategies for Literacy Independence across the Curriculum (SLIC), in 
eight middle and high schools throughout the District. An additional eight middle and 
high schools are participating as comparison schools. In total, 16 middle and high schools 
are participating in the San Diego Striving Readers evaluation study. This report 
summarizes the targeted and whole-school SLIC literacy programs and presents 
implementation and impact findings from the first four years (2006-07, 2007-08, 2008- 
09, 2009-10) of the 5-year evaluation study. 


Overview of SLIC 


The targeted intervention is a supplemental class that replaces an elective for eligible 7th 
through 10th grade students. The supplemental class is taught by teachers trained in the 
Strategies for Literacy Independence across the Curriculum (SLIC) model, developed by 
researchers from New Zealand (Trevor McDonald and Christina Thornley).! The 
professional development-based model is not a prescribed curriculum, but is rather a set 
of literacy strategies developed to enhance students’ skills in reading and writing. 
Students are taught how authors use different text forms to present particular types of 
information and how the surface features of a text convey information about the content 
of the text. Use of the model and strategies by teachers is informed by periodic 
administrations (every 2-3 months) of the SLIC assessment (developed jointly by the 
SLIC developers, the SDUSD project leadership, and researchers at the University of 
California, Berkeley (UCB)/Berkeley Evaluation and Assessment Research Center 
(BEAR), which is closely aligned to the SLIC model. The targeted intervention class uses 
a variety of persuasive, expository, and narrative texts, including grade-level textbooks 


' The amount of class time devoted to SLIC varies by school site, and according to the schedule 
established by each school. Presentation of the average amounts of SLIC class time for Years 1-2 
are found in Appendix F. 


used in core content areas (Social Science, Science, Mathematics, and Language Arts) in 
the secondary school, magazine and newspaper articles, short stories, and novels.” 


In the targeted SLIC intervention, students who meet specific eligibility criteria are 
randomly assigned into targeted SLIC intervention classes or into the control condition 
(regular elective class). Students in 7th through 10th grade are eligible for random 
assignment into the targeted SLIC intervention class if they are 1) reading at least two 
years below grade level, as measured by the Degrees of Reading Power (DRP) 
assessment at the end of the prior school year, 2) are reading at a ‘basic’ level or below, 
as measured by the California Standards Test—English Language Arts (CST-ELA) test, 
or 3) are labeled ‘intermediate’ or below on the California English Language 
Development Test (CELDT). 


The whole-school intervention is based on the same SLIC literacy strategies used in the 
targeted SLIC intervention. SLIC developers and leadership, along with SLIC coaches 
located at each of the schools, provide professional development to content-area teachers 
in teaching the literacy strategies. The whole-school professional development is 
introduced gradually to content-area teachers, in order to build momentum and increase 
teacher buy in. All sixth-grade through twelfth-grade students in the intervention schools 
taught by teachers trained in the whole-school intervention can receive instruction. 


In Year 2 (2007-08), for the targeted SLIC intervention evaluation, there were a total of 
758 eligible students assigned to the SLIC intervention class, including 53 students 
continuing from Year 1, and 831 eligible students assigned to the control group, 
including 55 students continuing from Year 1. 


In Year 3 (2008-09), there were 566 eligible students with continuing assignments to the 
treatment group and 324 new assignments. In the control group, there were 615 
continuing assignments and 314 new assignments. In addition to students who met 
eligibility criteria to participate in the treatment class, there were students in both Years 2 
and 3 who no longer met these criteria, but remained in the analytic sample on the basis 
of previous assignments to the treatment group or the control group. For the whole-school 
SLIC intervention evaluation, a total of 16,895 students (6,809 students in intervention 
schools, 10,086 students in control schools) participated in the 16 study schools. 


* SLIC instruction included the use of grade level textbooks and expository, persuasive and 
narrative text from magazines, newspapers and books. Additionally, new curriculum materials 
were created for Year 3. 


In Year 4 (2009-10), there were no new random assignments, rather students continued in 


their current assignments. There were 734 students with continuing assignments in the 
targeted SLIC treatment group and 735 students with continuing assignments to the 


control group. In comparison to the previous year, the number of students participating 
in the whole-school SLIC intervention decreased slightly. In Year 4, there were 16,310 
students (6,647 students in intervention schools, 9,663 students in control schools) who 
enrolled in the 16 study schools. 


Targeted SLIC Intervention 


Implementation of the Targeted Intervention 


In general, there has been growth in the fidelity of implementation of the targeted 


intervention model. 


The hours of professional development decreased from Year | to Year 2, 
increased from Year 2 to Year 3, and varied widely in Year 4, ranging from 10 to 
258 hours, and averaging 131 hours for new intervention teachers and 88 hours 
for experienced intervention teachers. Overall, the amount of PD was lower in 
Year 4 than in prior years. Program developers focused most intensively on the 
targeted intervention in Year | and held large cross-site events for both the 
targeted and whole school programs. They devoted more time to the whole- 
school program in Year 2, with considerable time on site specific work. In Year 
3, developers worked more directly with targeted intervention teachers than in 
prior years and offered more individualized guidance. In Year 4, leadership saw 
less need for professional development with experienced teachers, and focused 
their efforts on new intervention teachers. 


The program’s delivery of professional development changed from Year 1 to 
Year 4. In Years 1 and 2, professional development (PD) was provided to 
coaches and teachers, and it was expected that the coaches would provide site- 
based support to the targeted intervention teachers. Over the course of four years 
the program developers came to believe that coaches should have experience 
enacting the instructional model before coaching others in its implementation, and 
a former intervention teacher became a coach in Year 4 and at other sites the roles 
of teachers and coaches overlapped. 


The number and usability of material supports increased greatly from Year 1 to 
Year 3 through the provision of program documents, research reports, assessment 
rubrics and, in Year 3, curriculum units for teachers. Documents were updated for 
Year 4, but no new materials were introduced. As a result of the increased 
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supports, the model became more fully and clearly articulated over the four years 
of the program. The curriculum units address the steep learning curve for new 
SLIC teachers, and speak to longer term issues of program sustainability. 


In Year 4, all schools achieved medium fidelity to the classroom implementation 
model. The proportion of classrooms achieving low, medium, and high fidelity of 
implementation was unchanged from Year 3 to Year 4. 


In Years 2 and 3, there was at least as much difference in classroom fidelity 
within schools as there was between schools, and “medium” fidelity often 
reflected the averaging of high and low implementation across classrooms at a site 
rather than medium fidelity in all classrooms. By Year 4, there appeared to be 
somewhat more consistency across classrooms, although the pattern of higher and 
lower implementation within schools remained to some degree. 


Evidence suggests there was greater growth in implementation fidelity in middle 
schools than high schools between Year 2 and Year 3. The evidence in Year 4 is 
mixed. Differences may be associated with extrinsic factors, such as the turnover 
of personnel (principals, coaches, teachers) and these varied by program year. 


Turnover of personnel from Year | to Year 4 presented a particular challenge for 
the program, which relies on trained teachers. The proportion of continuing 
teachers and coaches was higher in Year 3 and Year 4 than in prior years. In Year 
2, due to teacher and coach turnover and the addition of three additional schools, 
89% of targeted intervention teachers were new to the program as were 66% of 
coaches. In Year 3, 25% of teachers were new to the program, but all coaches 
remained, although one took on a non-coaching role. In fall of Year 4, 69% of 
Year 4 teachers were continuing intervention teachers. However, coach turnover 
was high. Two high schools and one middle school had no coach in Year 4, while 
two middle schools shared a coach and one high school had a new coach, a former 
SLIC teacher. 


Two schools had combined SLIC/ESL classes and there was a greater focus on 
student talk in Year 3 in an effort to address instructional needs for English 
Learners. In addition, efforts were made to reach EL students through 
differentiated instruction. Otherwise, the program model maintained a consistent 
approach to literacy for all students. While targeted intervention teachers and 
coaches generally stated that SLIC is helpful for EL students, several expressed 
the view that SLIC alone is not sufficient. 


Asked about strengths and weaknesses of SLIC, several teachers mentioned the 
support they receive from program developers, leadership and coaches, the focus 
on skills, and the goal of student independence as strengths. A commonly named 
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weakness was repetitiousness and an insufficient range of strategies to foster 
student engagement. Some stated that, in Years 3 and 4, engagement was a 
greater problem among students who had remained in the program for a second or 
third year, and classes that included a number of these students were considered 
challenging to teach. 


Two confounds were noted. The first is that the whole school program instructs 
students in SLIC strategies appropriate to each content area, and that control 
students are therefore exposed to the same program as SLIC students although to 
a much lesser degree. This may make differences in student gains harder to 
detect. A second confound is that some control students may be in supplementary 
literacy classes with teachers who have participated in whole school PD, which 
would, if true, further muddy the distinction between SLIC and control 
conditions. 


Implications. Given these findings, greater impacts might be anticipated in Year 
3 than in Year 2, particularly in higher implementation classrooms. While some 
aspects of implementation (differentiation, coverage of the curriculum) increased 
in Year 4, there might be lower impacts simply because there were no newly 
assigned students in this year. Since students can learn the same SLIC skills and 
strategies in haa grade as in 10" and since they are exposed to the full SLIC 
program at each grade level, there is no reason to anticipate that students 
remaining in the program longer will perform better than those testing out after a 
year, except as a result of additional practice and exposure to a more fully 
implemented program. Stated differently, if the program is effective, it is 
reasonable to expect that one or two years of exposure to the program will be 
sufficient for students. 


Targeted SLIC Impacts on Students 


No significant differences were found between students in the targeted SLIC 
intervention and control groups on any of the outcome measures: the Degrees of 
Reading Power (DRP), the California Standards Test — English Language Arts 
(CST-ELA) scores, California High School Exit Exam - English Language Arts 
(CAHSEE-ELA), or student motivation. This held true after controlling for 
various covariates, including the students’ pre-test score (as applicable), students’ 
gender, a proxy for socio-economic status, students’ grade level, and the students’ 
English learner status. This also held true for students who participated in the 
targeted SLIC intervention for two years, as well as for students who participated 
for three full years. 


No differences between students in the targeted SLIC intervention and control 
groups were found when examining students’ performance by subgroups of 
interest (e.g., English learners, students in middle schools, and students in high 
schools). 
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e Using the Bloom’s adjustment, for the DRP, there was a targeted SLIC effect for 
one year and two years of participation, but not for three years of participation. 
For the CST-ELA, there was no SLIC effect for students regardless of years of 
participation. 


e Using the Bloom’s adjustment, there was no targeted SLIC effect for any of the 
subgroups of interest (e.g., English learners, students in middle schools, students 
in high schools), regardless of years of participation. 


Whole-School SLIC Intervention 


Implementation of the Whole-School Professional Development Model 


e The amount of individual support to teachers and content-specific professional 
development increased from Year | to Year 2, and more content teachers were 
involved in sustained work with program personnel in Year 2. At the same time, 
the amount of cross-site and “whole-school” professional development (PD) 
declined from Year | to Year 2. In Year 3, there was a substantial increase in the 
amount of support provided to content-area teachers. While the amount whole- 
school PD continued to decline at most sites, the amount of content-specific PD 
and individual support generally increased. In Year 4, there was an overall 
decline in PD following the departure of five coaches. Sites with continuing 
coaches had high levels of PD, but there was considerable variation across sites. 


e Surveys suggest that there was more interest in the program than developers and 
coaches were able to meet in Year 2. The increase in support to content-area 
teachers in Year 3 was made possible by a significant shift in roles, in which 
coaches assumed an increased responsibility for content-area work. In Year 4, 
coaches were the primary providers of content area PD. 


e Across the first three program years, the greatest interest in program participation 
was expressed by Science teachers. In Years 2, 3, and 4 English teachers indicated 
increased interest in the program, perhaps facilitated by increased attention in the 
Year 2 program to narrative text. In Year 4 there was also greater interest among 
social studies teachers. While math teachers have generally shown the lowest 
interest in the program, their interest increased in Year 3, and some math teachers 
began to work intensively with program coaches. 


e Resistance to the SLIC program occurred at some schools. However, resistance 
was less pronounced in Year 2 at the schools continuing with the program, while 
new challenges were faced at the three new schools. With all but one school 
continuing from Year 2 to Year 3, and all schools continuing from Year 3 to Year 
4, the ability of the remaining coaches to work with content-area teachers was 
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supported by experience at their sites. The proportion of continuing teachers 
climbed from 37% in Year 2 to 76% in Year 3 and 81% in Year 4. 


e Across all years, challenges to implementation included levels of site leadership 
support at some schools, and different sources of staff resistance at a few schools. 
In Year 2, development of the SLIC assessment took large amounts of time, 
posing a challenge to full implementation of the whole-school intervention by 
district leadership, developers, and coaches. In Years 3 and 4, the SLIC 
assessment demanded considerably less time. 


e The program developers were present more in Year 2 than Year 1, the 
professional development materials were more extensive and better developed by 
Year 3, and for the most part the coaches remaining with the program deepened 
their connections with the school sites, which facilitated their work. With a shift 
of more content-area work to coaches in Year 3, availability of program staff to 
content teachers was less an issue and support was stronger. This support 
diminished in Year 4 with the departure of some coaches, although the inclusion 
of targeted intervention teachers who were well integrated at the sites and who 
served in professional development roles allowed some continuity in the 
professional development at those sites. 


e In Year 2, there was greater interest and participation in the program in high 
schools than in middle schools. This difference was reflected in interviews, 
surveys, and records of professional development participation. This difference 
may have been due to the greater stability of program staffing at high schools 
between years | and 2. In Years 3 and 4, this difference was less evident. 


Implementation of the Whole-School Classroom Model 


e Classroom implementation was not quantitatively measured in Year 1, although 
implementation was assessed through qualitative measures to be low both in 
fidelity and level of implementation. 


e Although there were no comparable measures in Years | and 2, it is likely that 
classroom implementation expanded in Year 2 but was not as robust as leadership 
anticipated it might be. Using data from survey response, in Year 2, 13% of 
intervention schools met the “expected” level of implementation, and 50% of 
schools approached that level. However, this rating reflects the percent of 
teachers who “made specific plans” to implement SLIC (the percent who “tried 
it,” according to coaches, was similar). The percent of teachers and schools where 
the curriculum was incorporated in content-area instruction in an ongoing manner 
was far lower, and student surveys reflected no change in instruction in the 
content areas. 


e In Year 3, content teachers’ self-reported use of in-class reading increased. 
However, while self-reported instruction in program-related literacy strategies 
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was moderately high, these did not generally increase from Year 2 to Year 3 and 
decreased in Year 4. Classroom observations suggest that some content-area 
teachers have clearly taken on the program model, while others adhere to other 
models of literacy instruction. This may be a particular issue in English 
classrooms, where teachers may have stronger opinions about literacy instruction. 


Based on surveys and interviews, there was higher implementation of the 
classroom model in high schools than in middle schools in Year 2. At the same 
time, there was greater confidence among middle school content teachers in their 
literacy instruction, and a somewhat different interpretation of the program at the 
middle and high school levels. In Year 3 and 4, these differences were again less 
evident. 


There was a decline in implementation at some sites and an increase at others 
although, overall, there was a slight decline from Year 3. Since a relatively high 
proportion of content teachers remained at their schools, their cumulative 
exposure to the program increased. 


Evidence regarding differences between intervention and comparison schools in 
the level of literacy practice in Year 4 are mixed, with classroom observations 
indicating higher levels of literacy practice at intervention schools, student 
surveys suggesting higher levels at comparison schools, and teacher surveys 
showing more literacy practice at comparison schools in Year 2, at intervention 
schools in Year 3, and similar levels in Year 4. 


Whole-School SLIC Impacts on Students 


In Year 4, there was no significant difference between the total sample of students 
in the whole-SLIC school intervention (intervention schools) and their control 
counterparts (comparison schools) on the CST-ELA, DRP, CAHSEE-English 
Language Arts or in their motivation levels. This held true even after controlling 
for various covariates, including the students’ pre-test score (as applicable), 
students’ gender, a proxy for socio-economic status, students’ grade level, and the 
students’ English learner status. 


Promising effects of the whole-school SLIC program were found with high school 
students. For example, when examining subgroups of students, significant 
differences were found between high school students in the whole-school SLIC 
intervention group (intervention schools), compared to students in the control 
group (comparison schools) on the Degrees of Reading Power (DRP), but not 
with middle school students or with English learners. This held true even after 
controlling for various covariates, including the students’ pre-test score (as 
applicable), students’ gender, a proxy for socio-economic status, students’ grade 
level, and the students’ English learner status. 
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e There were no significant differences found on the CST-ELA and CAHSEE 
between students in the whole-school SLIC intervention and their control 
counterparts when examining students’ performance by subgroups of interest 
(e.g., English learners, students in middle schools, students in high schools). 


e Middle school students participating in the whole-school SLIC program 
(intervention schools) also showed significantly higher levels of student 
motivation as compared students in the control groups (comparison schools). 
Similar trends were not found with high school students or with English learners. 
This held true even after controlling for various covariates, including the students 
pre-test score (as applicable), students’ gender, a proxy for socio-economic status, 
and students’ grade level. 


> 


Conclusion 


While the challenge for the leadership and developers in Year | was too little time in an 
instructional period shortened by delays in program start-up, the challenge in Year 2 was 
the rapidly expanded scale of the program and, within that context, meeting the varied 
and competing needs of the targeted SLIC program, the whole-school SLIC program, and 
the assessment work. This situation was exacerbated by staff turnover from Year | to 
Year 2, and the need to train new teachers, some as new to teaching as they were to 
SLIC. In Year 2 there was not, at some schools, a core of experienced SLIC teachers 
who could help guide new SLIC teachers during this period of rapid scale-up and 
competing program demands. By the end of Year 2, however, the whole-school SLIC 
program had gained momentum through the work of developers, leadership, and coaches, 
and there was substantial interest in the program among content-area teachers. 
Additionally, there was a solid base for further implementation of the targeted 
intervention in the great majority of intervention schools in Years 3 and 4. 


In terms of student impacts, the results were somewhat promising. Students in the 
targeted SLIC intervention scored higher on the Degrees of Reading Power (DRP) than 
control group students after one year in the program, however, this difference was not 
found among subgroup analyses of English learners, middle school students, or high 
school students. Further, for students who participated for two or three years in the 
targeted SLIC program, there was no difference between participation in the intervention 
class and belonging to the control group. 


In Year 4, for the targeted SLIC impact evaluation, no differences were found between 
students who participated in the targeted SLIC program (treatment group) and their 
counterparts who did not (control group) on multiple outcome measures, such as the 
Degrees of Reading Power (DRP), California Standards Test — English Language Arts 
(CST-ELA) scores, California High School Exit Exam — English Language Arts 
(CAHSEE-ELA), or student motivation. This was the case even after controlling for key 
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covariates (e.g., parent education or percent with BA degree as a proxy for 
socioeconomic status, pre-test, English learner status), by subgroups, and regardless of 
years of participation in the targeted SLIC program. 


Using the Bloom’s adjustment (TOT analysis), there was a small targeted SLIC effect on 
the DRP for one year (p<.05) and two years of participation (p<.05) but not for three 
years of participation. For one year and two years of participation the percentage effect of 
the intervention is 3.12% and 4.46%, respectively. There was no TOT effect on the CST- 
ELA. 


For the whole-school SLIC impact evaluation, significant differences were found on the 
Degrees of Reading Power (DRP) between high school students in the whole-school 
SLIC intervention group (intervention schools) compared to students in the control group 
(comparison schools) (p<.05), but not on the California Standards Test- English 
Language Arts (CST-ELA). Significant differences were also found with comparison 
middle school students having higher student motivation scores than intervention school 
students (35.7 vs. 34.5, p <.01). Overall, the impact findings suggest some positive, 
albeit mixed, findings for the targeted and whole-school SLIC programs. 


Il. INTRODUCTION AND STUDY BACKGROUND 


The San Diego Unified School District Striving Readers Project aims to work within the 
research base on adolescent literacy to conceptualize, describe and apply the elements of 
effective teaching and professional development with the goal of increasing the literacy 
achievement of struggling adolescent readers. 


Context for the Study 


The San Diego Unified School District currently enrolls approximately 132,000 students 
in 228 schools, and has provided more than 150 years of service to San Diego’s children. 
It is the second largest district in California, and the eighth largest urban district in the 
United States. The student population is extremely diverse representing more than 15 
ethnic groups and over 60 languages and dialects. About 30 percent of the district’s 
students are English learners. Over half (59%) of the district’s students are eligible for 
federal free and reduced price lunch program funding and 12 percent of the district’s 
students are identified as students with special needs. SDUSD operates 28 high schools 
and 24 middle /junior high schools. 


The intervention and comparison schools participating in the SDUSD Striving Readers 
program reflect the diversity of the district. The intervention schools were somewhat 
more socio-economically disadvantaged (74-100% receive free or reduced price lunch) 
and had higher percentages of English learners (12-44%) than the comparison schools 
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(29-87% receive free or reduced price lunch; 10-43% are designated as English learners). 
Interestingly, since the start of the Striving Readers study in 2006-07, the percentages of 
students qualifying as socio-economically disadvantaged and who are designated as 
English learners have increased annually overall. Both types of schools serve comparable 
percentages of students with disabilities (7-19% for intervention schools, 8-20% for 
comparison schools). In addition, while most of the intervention and comparison schools 
experienced growth in their California Standards Test scores over time, much of this was 
a District-wide phenomenon as well (San Diego Unified School District, School 
Accountability Report Cards, 2005-06, 2006-07, 2007-08, 2008-09). 


It should be noted that intervention middle schools and high schools were selected as 
feeder groups so students could be followed longitudinally over the course of the 5-year 
study. This was not possible for all of the comparison schools. Intervention and 
comparison schools were selected based on 1) demographic and educational achievement 
indicators; 2) school leadership and staff opinion that the intervention was compatible 
with articulated site goals and plans; and 3) number of students in the school with larger 
numbers helping to address sample size (power) issues. 


Theoretical Rationale for and Description of the Intervention Models 


The Striving Readers literacy model in the San Diego Unified School District (SDUSD) is called 
Strategies for Literacy Independence across the Curriculum (SLIC). The instructional and 
curricular approach is substantially the same for targeted SLIC intervention and whole-school 
(school-wide) content-area classes with the primary difference being intensity of instruction, or 
dosage. The targeted SLIC class replaces an elective class for most eligible 7th through 
10th grade students. 


Following is a brief description of each model, including its theoretical underpinnings, 
along with a description of the students participating in each intervention, the 
participating schools and grades, and number of students served. 


Targeted SLIC Intervention 


The Strategies for Literacy Independence across the Curriculum (SLIC) professional 
development model relies on the transmission of SLIC literacy knowledge from program 
developers/leaders to coaches and intervention teachers and from teachers to students, 
and ongoing coach support for teachers. It is not a prescribed curriculum, but is rather a 
set of literacy strategies developed to enhance students’ skills in reading and writing. 
Based on research conducted by McDonald & Thornley (Thornley & McDonald, 2002; 
McDonald & Thorley, 2004; McDonald & Thornley, 2005) and supported by ongoing 
reviews of the adolescent literacy research base, SLIC is premised on the idea that in 
order to make meaning students must understand the ways authors use different text 
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forms to present particular types of information and how the surface features of a text 
convey information about the content of the text. The SLIC program is also premised on 
a strong belief that the best path to increased adolescent literacy achievement is through 
building the knowledge base of teachers in the ways texts work and their pedagogical 
knowledge of teaching about texts and analyzing student work to plan instruction. 


The starting point is instruction in how to use text features and the structure of texts to 
support meaning-making and progresses to strategic reading behaviors to achieve deeper 
understanding across a range of challenging texts and tasks. In the targeted SLIC class, 
students are given explicit instruction in recognizing and using text features such as titles, 
subtitles, captions, font style, and graphics to gain a rapid understanding of a text’s 
content, purpose, structure, and organization as expository, persuasive, or narrative text. 
In addition to previewing text prior to reading, students receive explicit instruction in 
other strategic reading behaviors such as cross-checking among text features and between 
text features and running text to verify their understanding, using contextual clues to 
apprehend unfamiliar vocabulary, note-making and other forms of writing to organize 
textual information derived from readings, and breaking writing prompts into component 
questions. The model assumes that, over time, students actively learn about text features, 
forms, and structures and gradually build independence in using these through scaffolded 
instruction and independent reading and writing practice.” 


Use of the targeted SLIC model and strategies by teachers is informed by periodic 
administrations, every 2-3 months, of the SLIC assessment (developed jointly by the 
SLIC developers, the SDUSD project leadership, and researchers at UC Berkeley), which 
is Closely aligned to the SLIC model, along with the California English-Language Arts 
Content Standards. The targeted intervention class uses a variety of persuasive, 
expository, and narrative texts, including textbooks used in core content areas (social 
science, science, mathematics, and language arts) in the secondary school, magazine and 
newspaper articles, short stories, and novels. 


Striving readers in the 7th through 10th grades are eligible for random assignment into 
the targeted SLIC intervention class if they meet at least one of the following eligibility 
criteria: 1) reading at least two years below grade level, as measured by the Degrees of 
Reading Power (DRP) assessment at the end of the prior school year; 2) reading at a 
‘basic’ level or below, as measured by the California Standards Test—English Language 
Arts (CST-ELA) test; or 3) are labeled ‘intermediate’ or below on the California English 
Language Development Test (CELDT). Targeted SLIC is a multi-year intervention, 
which students can enter in the 7th grade or beyond and remain through grade 10 unless 
their test scores show they no longer qualify for the intervention. 


> SLIC instruction does not rely on the use of technology. 
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Student outcomes for the targeted intervention are pooled across study years based on the 
number of years students have valid random assignments to treatment or control 
conditions. In Year 4, students who remained eligible for the SLIC program retained their 
random assignments, while others may have tested out and had the opportunity to exit the 
targeted SLIC classes. For a one-year-of-participation model, first year outcomes for all 
students with valid random assignments are included, totaling 1,115 treatment-assigned 
students and 1,151 control-assigned students across three years. For a two-years-of- 
participation model, students were included if they began in Year 1, 2 or 3 and continued 
in a participating school and grade for an additional year, for a total of 929 treatment- 
assigned students and 903 control-assigned students. For additional details, see “Pathway 
to Analytic Samples” below. 


Pathway to Analytic Sample 

Based on changing enrollment and incoming test scores, schools request updated 
information on students eligible for random assignment at multiple time points, varying 
by school, prior to the beginning of the school. In addition, in the late summer and early 
fall, schools assess an unknown number of newly enrolling students for eligibility, 
contacting the evaluation team for random assignments as needed. As such, there is no 
baseline assessment against which later removals (e.g., due to test scores rising above 
eligibility cutpoints prior to treatment) can be defined. Therefore, the numbers of students 
assessed (see Figures | -3) refer to the number of students enrolled in the fall of each 
study year, while assessment refers to assessment based on the data available at the 
beginning of the given study year. 


For each study year, data are gathered on total enrollment at participating schools and 
grades. Across the first four study years, this enrollment totaled 10,295 students, with 
students included in this total more than once if in a participating school and grade more 
than one year. If students had a random assignment from a previous study year—.e., for 
Year 4 from Years 1, 2 or 3 —they could not be given a new random as signment.* The 
remaining students were assessed for eligibility for random assignment. If their most 
recent test scores did not meet the study’s eligibility criteria, if they had certain types of 
special education placements, or if they were identified as a beginning English learner 
enrolled in the district for less than one year, they were not eligible for random 
assignment.” As noted earlier, in Year 4, no new random assignments were made. 
Eligible students continued in their original assignment. 


* However, Year 1 control students were randomly re-assigned to treatment and control in Year 2. 
Approximately 30 continuing control students therefore ended up in the treatment class. 

> In general, the probability of assignment to the treatment or control conditions was equal, although, in 
some cases, the treatment-control ratio was adjusted to meet a specific site’s need for fewer or more 
students in the treatment class. 
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The remaining students were technically eligible for random assignment to treatment or 
control conditions. Three conditions led to eligible students not receiving valid random 
assignments. First, because of low compliance with random assignment in Year 1 of the 
study, assignments from three of five participating schools were not treated as valid and 
were excluded from all analyses. Students enrolled in the treatment class at these three 
schools were kept in the class in subsequent years if they continued to meet eligibility 
criteria, but were excluded from all analyses. Students not enrolled in the class at these 
schools were given new random assignments in subsequent years if they met eligibility 
criteria. Second, beginning in Year 2, schools were provided the opportunity to exempt 
eligible students from random assignment prior to the making of assignments. Schools 
made these exclusions for a variety of reasons, including the desire to enroll students in 
alternate interventions and the belief that low test scores were not an accurate reflection 
of a given student’s ability. Finally, a substantial number of eligible students did not 
receive any random assignment due to post-assignment changes in students’ eligibility or 
enrollment. These students are labeled “not assigned.” The remaining students had valid 
random assignments beginning in one of the three study years. 


The models for one to three years of participation in the targeted SLIC program are 
combined across program cohorts (2006-10). For example, one year of participation in 
the targeted SLIC intervention (see Figure 1) includes students given assignments in all 
four study years. Because we begin with total enrollment and the most current study data 
for each given study year, no follow-up exclusions are necessary for a model of one year 
of program participation. The model for two years of participation (see Figure 2) begins 
with the first two study years and, as a last step, excludes students not enrolled in a 
participating school or grade for a second year. The same applies for the three-year 
participation model (see Figure 3). Following an intent-to-treat model, students with 
valid initial assignments were kept in the two-year or three-year model whether or not 
they continued to meet program eligibility criteria. However, in light of the fact that 
many treatment-assigned students test out of the program and are removed from the 
treatment class after their first year of participation, treatment-on-treated (TOT) effects 
are included in this report. 
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Combined Enrollment, Years 1-3 
(n=10,295) 


Previous Assignments (n=1,304) 


Assessed for treatment-628 
7 Eligibility control-676 
Ineligible (n=8,991) 


(n=5,318) 
test scores-5,036 
special education-24 


EL status-41 
Eligible for Random 
: Assignment ; 
Not Assigned (n=3,673) Exempted 
(n=800) (n=110) 
Excluded Post-RA* 
(n=497) 
Randomly Assigned 
(n=2,266) 
Assigned to treatment group Assigned to control group 
(n=1,115) (n=1, 151) 


Included in analysis as control 


Included in analysis as treatment 


(n=1,115) (n=1,151) 


* Random assignment was not effectively implemented at 3 of 5 Year | schools. Students from these schools were dropped from the 
targeted intervention impact analysis. 


Figure 1. Random Assignment of Students into Targeted SLIC Intervention, One 
Year of Participation 
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Combined Enrollment, Years 1-3 
(n=10,295) 


Previous Assignments (n=1,304) 


Assessed for treatment-628 
F: Eligibility control-676 
Ineligible (n=8,991) 


(n=5,318) 
test scores-5,036 
special education-24 
EL status-4 1 


Eligible for Random 


Assignment 


Not Assigned _ Exempted 
(n=800) reer) (n=110) 
Excluded Post-RA* 

(n=497) 


Randomly Assigned 
(n=2,266) 


Assigned to treatment group 
(n=1,115) 


Assigned to control group 
(n=1,151) 


Excluded due to non-enrollment 


in second year (n=248) 


Excluded due to non-enrollment 
in second year (n=186) 


Included in analysis as treatment Included in analysis as control 
(n=929) (n=903) 


* Random assignment was not effectively implemented at 3 of 5 Year | schools. Students from these schools were dropped from the 


targeted intervention impact analysis. 


Figure 2. Random Assignment of Students into Targeted SLIC Intervention, Two Years of 


Participation 
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Combined Enrollment, Years 1-2 
(n=5,910) 


Previous Assignments (n=123) 
treatment-63 
control-60 


Assessed for 
Eligibility 
(n=5,787) 


Ineligible 
(n=3,141) 
test scores-3,003 
special education-1 12 
EL status-26 


Eligible for Random 
Assignment 
(n=2,646) 


Not Assigned 
(n=478) 


Exempted 
(n=59) 


Excluded Post-RA* 
(n=497) 


Randomly Assigned 
(n=1,612) 
Assigned to treatment group Assigned to control group 

(n=773) (0-839) 


Excluded due to non-enrollment 
in second year or third year 
(n=404) 


Excluded due to non-enrollment 
in second year or third year 
(n=321) 


Included in analysis as treatment Included in analysis as control 
(n=452) (n=435) 


* Random assignment was not effectively implemented at 3 of 5 Year | schools. Students from these schools were dropped from the 
targeted intervention impact analysis. 


Figure 3. Random Assignment of Students into Targeted SLIC Intervention, Three Years of 
Participation 


Whole-School SLIC Intervention 


The whole-school intervention is based on the same SLIC literacy strategies used in the 
targeted SLIC intervention. The SLIC developer and SLIC coaches provide professional 
development to content-area teachers in teaching the literacy strategies. The whole- 
school professional development is introduced gradually to content-area teachers in order 
to build momentum and increase teacher buy in. Priority is given to teachers serving 
students who are also attending the targeted SLIC classes and to those teachers who 
express the most interest in being trained. In Year 1, these were often Science teachers, 
reflecting the model’s focus on expository text. In Years 2 and 3, implementation of the 
whole-school SLIC program was scaled up considerably, although the content areas of 
trained teachers and the extent of training varied across schools. In Year 4, additional 
content teachers were trained at the schools that had coaches or SLIC teachers who took 
on a professional development role, and implementation continued at some level 
elsewhere. 
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All sixth-grade through twelfth-grade students in the intervention schools taught by 
teachers trained in the whole-school intervention can receive instruction. Over time, the 
number of students in the whole-school intervention has remained generally constant as it 
involves potentially all students in the participating schools. In the second year of the 
study (2007-08), there were 16,627 students (6,498 students in intervention schools, 
10,129 students in control schools) enrolled in the 16 study schools. In the third year of 
the study (2008-09), there were 16,895 students (6,809 intervention, 10,086 control). In 
the 4" year of the study, there was a total of 16,310 students (6,647 intervention, 9,663 
control) in the study. 


Logic Model 


See combined Targeted SLIC and Whole-School SLIC Logic Model (Figure 4.) 
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Figure 4. SDUSD: Striving Readers' Logic Model for Implementation Process and Fidelity of Implementation 


Skills & Abilities (Surveys & 
Observations) 


1 SLIC Classes, (students 2 or more years below grade 
level on 1 or more of 3 assessments) 


Leadership 1 Leaders and developers build teacher 
knowledge and understanding of striving readers 
curriculum (scope & sequence) in intervention 


classes to SLIC teachers and coaches. 


District Leadership 
Daily explicit instruction in strategic reading behaviors from 
scope & sequence, moving from overview of content using 
text features to reading and meaning-making in running 


Increased reading & writing 
stamina, independence, efficacy, 
use of text, academic language, 
writing for learning 


Program Developers 
2 Leaders and developers develop planning 


School-Level 
Literacy Coaches 


Intervention 
Teachers (SLIC) 


Content-area 
Teachers (school 
wide) 


sequence, curriculum & pedagogical practice of 
SLIC teachers and coaches through 
observation, collaborative planning, coaching, 
team teaching, demonstration, review of student 
work, reflection and feedback. PD scaffolds 
developed & PD roles & delivery provided 
according to teacher/classroom needs 


3 Leaders and developers build content-area 
teachers’ and coaches’ knowledge of the 
application of the scope & sequence to content- 


text, cross-checking for accuracy and higher level skills of 
inference, synthesis and critique to building vocab 
knowledge 


Daily reading in increasingly complex texts to complete 
increasingly complex grade level written tasks 


Curricular emphasis responsive to literacy demands of 
district testina. school proarams. student need 


Measures (Diagnostics & 


Standardized) 


Targeted SLIC Intermediate: 
All students make progress on 
writing benchmarks (four point 
scale) and on two reading 
diagnostics during the year 


2 SLIC Instructional approach focusing on literacy skills in 


area (school-wide) classroom instruction 
order to access content 


Targeted SLIC Long Term: 

All students make 2yrs+ growth per 
year on DRP 

All students make yearly progress 
of at least one sub level to at least 
proficient by 10" grade 

ELs increase OPL by one 
proficiency level per year until 
advanced or reclassified (CELDT) 
All students score proficient or 
advanced on English 9&10 end of 
course exams (EEOCE) 


4 Leaders and developers develop planning 
sequence, curriculum and pedagogical practice 
Evaluation of content-area teachers (school-wide) and 
UCSD coaches through approaches described in (2) 
above. Coaches increasingly responsible for 
Oversight & supports to teachers. Coaching team provides 
Support professional development to middle schools; 
USDOE & Abt site-coaches offer ongoing support at three of 
four high schools. 


Teacher questioning to focus on metacognitive skills, build 
shared charts of learning processes 


Ongoing student reflection & review of progress, writing in 
academic journals and glossaries 


Material Resources 
Federal grant 
District infrastructure 


3&4 Content-area Teachers—School-wide (all SLIC, 
control & other students at intervention schools) 
Instruction in relevant SLIC strategies to increase 
interaction with text, preview text features to build 
knowledge base, read and note-make, write for academic 
purposes (at least three times weekly) 


All students (SLIC, control & others) 
5% increase in proficient or above 
each year (CST) 

5% reduction in students scoring 

5 Leaders and developers with UC Berkeley below or far below (CST) 

develop & trial diagnostic and summative All first time test takers pass 
curriculum assessments CAHSEE 


Assessment 


District Leadership 


5 Assessment 
Ongoing analysis of diagnostics and benchmark 
assessments to inform and differentiate instruction 


Train SLIC teachers and coaches in use of 
diagnostic assessments for ongoing planning 
and summative purposes 


Program Developers 


UC Berkeley (BEAR) 


Confounds and External Factors affecting Implementation: site support, teacher engagement, teacher & coach turnover, teacher & coach experience, literacy knowledge and beliefs, 
working relationships, literacy classes and supports to control group over and above content-area learning experiences 


Brief Overview of Key Evaluation Design Features 


Targeted SLIC Intervention 


The key research questions for evaluating the targeted SLIC intervention are: 


i, 


Will struggling readers assigned to the SLIC targeted intervention class improve 
their literacy outcomes, as measured by standardized tests and the project 
monitoring (SLIC) assessment, significantly more than struggling readers who do 
not receive the SLIC targeted intervention but who do receive the SLIC whole- 
school intervention? 


Will students in the SLIC intervention group be more likely to read at grade level, 
pass the California High School Exit Exam in 10th grade, enroll and successfully 
complete AP classes in eleventh/twelfth-grades, graduate from high school, and 
enroll in college than SLIC-eligible students who do not receive the SLIC targeted 
intervention but who do receive the SLIC whole-school intervention? 


Will struggling readers classified as English Learners in the SLIC intervention 
class improve their literacy outcomes compared to those classified as English 
Learners who do not receive the SLIC targeted intervention but receive the SLIC 
whole-school intervention? 


What is the fidelity and level of implementation of the targeted SLIC intervention 
in the participating intervention schools? What accounts for the variation? 


The impact of the targeted SLIC intervention is evaluated through a student-level, within- 
school randomized design to compare the achievement of students enrolled in the 


participating intervention schools using hierarchical linear modeling approaches. 
Students in grades 7 through 10 who are designated as eligible striving readers, based on 
designated criteria, are randomly assigned into SLIC targeted intervention classes or to 
the control group condition (whatever elective class the student would normally enroll 
in). For the duration of the study, targeted SLIC intervention students participate in the 
intervention classes for successive years, up through 10th grade, until they no longer 
meet eligibility criteria. 


In the San Diego Striving Readers project, students are the primary unit of analysis for 
the targeted and whole-school interventions. In Year 2 (2007-08), a continuing partial 
cohort of 7th grade students (from two schools) randomly assigned in the first year of the 
study matriculated into the 8th grade maintaining their current assignments and four new 
cohorts of students were randomly assigned at the 7th, 8th, 9th, and 10th grades. In Year 
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3 (2008-09), four new refresher cohorts of 7th, 8th, 9th and 10th graders were 
randomized to treatment and control groups. In Year 4 (2009-10), students retained their 
original random assignment and no new students were randomly assigned. In other 
words, students who remained eligible striving readers continued in their original 
assignment (treatment or control). Students who tested out of the targeted SLIC program 
were able to exit the program, but retained their original random assignment for analysis 
purposes. The eight cohorts will be combined, by grade, for analysis. Over time, the 
evaluation of impacts on students increasingly utilizes cross-sectional and longitudinal 
designs that can estimate effects of up to three years of targeted intervention on students. 


Impacts of the targeted SLIC intervention on instruction are examined using multiple 
sources of data, including pre/post classroom observations, semi-structured interviews 
with literacy coaches and intervention teachers, and teacher and student surveys. Fidelity 
of instruction to the treatment model was assessed using data from classroom observation 
instruments, interviews with literacy coaches, teachers, curriculum developers and 
program leadership, and teacher and student surveys. Fidelity of implementation of the 
professional development model was also assessed on the basis of records of teachers’ 
and coaches’ participation in professional development and other forms of training, 
interviews, and surveys. 


The measures used to examine student and teacher outcomes are: 

Degrees of Reading Power (DRP) 

California Standards Test (CST) 

California English Language Development Test (CELDT) 

California High School Exit Exam (CAHSEE) 

Striving Readers assessment (produced by SDUSD, Education Associates, BEAR) 
Classroom observation protocol 

Interview protocol 

Teacher surveys 


Student surveys 


Whole-School SLIC Intervention 


The evaluation of the whole-school SLIC intervention utilizes a hierarchical linear 


modeling approach to examine reading and other academic outcomes for all students in 
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intervention and comparison schools. A total of eight schools are participating in the 
whole-school intervention, while eight schools serve as comparison schools. The 
comparison schools represent a wider range of demographics than do the intervention 
schools. They do not implement the SLIC model, but implement the District’s regular 
language arts program. All students in the schools are included in the evaluation of the 
whole-school intervention. 


Impacts on instruction can be estimated using data from semi-structured interviews with 
literacy coaches, program leadership, classroom observations, and teacher surveys at 
intervention and comparison schools. 


Fidelity of instruction to the treatment model will be assessed using data from several 
sources: semi-structured interviews with literacy coaches, teachers and program 
leadership, observations of professional development sessions, coaches meetings and 
classrooms, and teacher surveys at intervention and comparison schools. Fidelity of 
implementation of the program model for professional development and support will be 
evaluated on the basis of records of participation. 


The key research questions for evaluating the whole-school SLIC intervention are: 


1. Will students attending schools that implemented both the whole-school and 
targeted components of the SLIC intervention program demonstrate more 
improvement in literacy skills, as measured by student scores on standardized 
assessments, than will students attending comparison schools that did not 
implement either component? 


2. Will the outcomes of students in schools that implemented both the whole-school 
and targeted components of the SLIC intervention program improve more each 
year over the course of the study, than will the outcomes of students attending 
comparison schools that did not implement either component? 


3. What is the fidelity and level of implementation of the whole-school SLIC 
intervention in the participating intervention schools? What accounts for 
variation in fidelity and implementation level? 


The measures used to examine student and teacher outcomes are: 
Degrees of Reading Power (DRP) 

California Standards Test (CST) 

California English Language Development Test (CELDT) 


California High School Exit Exam (CAHSEE) 
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Striving Readers assessment (produced by SDUSD, Education Associates, BEAR) 
Classroom observation protocol 

Interview protocol 

Teacher surveys 


Student surveys 


Il. EVALUATION OF THE IMPLEMENTATION OF THE TARGETED 
INTERVENTION: YEARS 1-4 


Summary of the Design 


The evaluation of the implementation of the targeted SLIC intervention drew on multiple 
sources of evidence, largely qualitative, including observations (of professional 
development sessions, weekly coach meetings in Years 1-3, and classroom instruction), 
annual interviews with key staff involved with the targeted SLIC intervention (SLIC 
program developers, program leadership, coaches, and intervention teachers), annually 
administered teacher surveys that examined literacy practices within classrooms, pre-post 
student surveys that tapped into students’ general experience in their literacy and other 
content-area classrooms, and document analysis. The research questions that guided the 
research in Years | through 4 were largely the same, although content-area research 
questions were added in Year 2. There were a few changes to the research process in the 
program’s second year and Years 2 and 3 saw the refinement of some research 
instruments. 


Surveys: Surveys questions were developed to align with the program model and further 
questions were adopted from existing validated instruments. The surveys were written 
and tested in Year |. A teacher survey and a student survey were given in Fall 2007, 
Spring 2008, Spring 2009, and Spring 2010 to all teachers and all students at 
intervention and comparison schools. Three questions were added to the Year 3 teacher 
survey to collect additional information on content area implementation. In Year 3, the 
response rate for the teacher survey was 71% for intervention schools and 75% for 
comparison schools; in Year 4 the intervention school response rate climbed to 84% with 
90% response at three schools. SLIC teachers’ response rate in Year 2 was 37% (7 of 
19), with 88% in Year 3 and 75% in Year 4 (this includes 12 of the 16 SLIC teachers who 
were present at any given time, and does not include two teachers who served as 
substitutes for a semester). 
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Interviews: All targeted SLIC teachers were interviewed in Years 1, 2, and 4, as were 
coaches and leadership. In Year 3, leadership and new teachers, departing teachers and 
coaches, and those assuming new roles were interviewed. Some additional questions 
were included in the interview protocol each year, with exit questions added in Year 3 
and interviews with principals added in Year 4. NVivo qualitative data analysis software 
was used to code the interviews and tally content of interest. 


Classroom Observations: An observation instrument was developed and first used in Fall 
2007. It was designed to allow collection of quantitative data about classroom 
implementation; this method is in contrast to the qualitative data collection in Year 1. 
Each targeted intervention class selected for observation was observed in fall, winter, and 
spring, and the classes to be observed were selected at random, with one class chosen for 
each teacher/grade level. There were between one and three researchers present at each 
observation. The observation instrument was subsequently revised, and the revised 
instrument used in winter 2007 and in spring 2008. It was revised again in spring of 2009 
and used in this form in 2010, and thus the observation data cannot be compared across 
all years, although observation data were comparable for winter 2007, spring of 2008 (in 
Year 2), and fall 2008, and spring 2009 (Year 3). 


Content area observations were conducted in Year 3 at the intervention middle (2) and 
high (2) schools reputed to have the highest implementation. In Year 4, three classrooms 
representing three content areas (English, history, science) were randomly selected for 
observation at all eight intervention schools. There were two sets of observations, in 
winter and spring, and classes were randomly selected each time. To parallel this, content 
classes in history, English, and science were observed at four of the comparison schools 
in winter and spring. Observation data were used in conjunction with interview and 
survey data. 


Professional Development and Coach Meeting Observations. Observations were 
conducted at professional development sessions, assessment scoring sessions, site visits 
by leadership, and meetings of the coaches and leadership (coach meetings were held in 
Years 1-3), and the observations were recorded in written notes. The research process 
associated with these observations did not change from Year | to Year 4. 


Document Collection. Observation at coach and professional development sessions 
included collection of program materials and copies of reading materials, such as 
magazine articles, when possible. Copies of readings assigned to students were also 
collected and the Flesch-Kincaid Grade Level formula was used to analyze the reading 
difficulty and approximate grade level of assigned readings, based on the word length and 
sentence length in the text. 
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Research Questions on the Implementation of the Targeted Intervention in Year 4 


What was the nature and amount of professional development/support for 
teachers/coaches in Year 4? 


Professional development/support for coaches 
What was the professional development model for coaches in Year 4? 


What was the variability (amounts) of professional development /support for coaches in 
Year 4? 


Do coaches’ understandings of the SLIC model conform to the views of the program 
leadership? 


What forms of direct support (e.g., district staff presence in teacher-coach meetings) and 
indirect support do coaches have in their work with SLIC teachers? 


What kind/amount of support is provided to coaches by site leadership? 

Professional development/support for SLIC teachers 

What types of support were provided to teachers? 

What types of PD were provided to teachers? 

What amount of PD was provided to teachers? 

What support/amount of support are teachers given in lesson planning and instruction? 
What support are teachers given in assessing student work to determine student needs? 
What kind/amount of support is provided by site leadership? 

What is the proportion of teachers receiving different levels of support? 


What was the fidelity and level of implementation, and the variability of classroom 
instruction in Year 4? 


What is the Year 4 classroom model (materials, resources, strategies, grouping, 
assessment practices, etc.)? 
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What proportion of teachers has access to program materials and resources? 
Fidelity of Implementation 


With respect to teaching points, how closely does SLIC teachers’ instruction follow the 
SLIC model? 


With respect to pedagogy, how closely does SLIC teachers’ instruction follow the SLIC 
model? 


How do SLIC teachers differentiate instruction to target students’ assessed needs? 


How do SLIC teachers make use of results from the program’s diagnostic assessments 
and student coursework? 


What types of reading materials do SLIC teachers use with students? (What 
content/academic content areas? Which text forms? Use of grade level texts?) 


What kinds of reading and writing tasks are students given in SLIC classes? 
Level of Implementation 


What proportion of teachers used instructional strategies, student groupings, instructional 
practices, assessment practices, etc. at different levels of implementation? 


What did the counterfactual (for targeted intervention) look like in Year 4? 


What were the experiences of the control students parallel to the interventions received 
by the treatment students? 
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Figure 5. Year 4 Data Sources on Implementation Linked with Research Questions: Targeted Intervention 


Research Questions Measures/Data Sources 
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seen Observations | Scoring Meetings Observations 
: Observations | held Y1-3 
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Professional development/support for coaches 
What was the professional 

development model for 

coaches in Year 4? 


What was the variability 
(amounts) of professional 
development /support for 
coaches in Year 4? 


What forms of direct support 
(e.g., district staff presence in 
teacher-coach meetings) and 
indirect support do coaches 
have in their work with SLIC 
teachers? 


What kind/amount of support 
is provided to coaches by 
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Do coaches’ understandings x x x x 
of the SLIC model conform to 

the views of the program 
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Figure 5. Year 4 Data Sources on Implementation Linked with Research Questions: Targeted Intervention (continued) 
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Figure 5. Year 4 Data Sources on Implementation Linked with Research Questions: Targeted Intervention (continued) 


Research Questions Measures/Data Sources 


Surveys | PD Site Visit Assessment Coach Classroom Record Review 
in Fall& | Observations | Observations | Scoring Meetings Observations 
i Observations | held Y1-3 
[2] 


[ 


Sep) 
Site Visit 


Teacher. 
So 
5 
Students|? 
mali 
If 
= 
aa 
laa 
P 
Attendance 


What is the Year4 classroom 
model (materials, resources, 
strategies, grouping, 

assessment practices, etc.)? 


What proportion of teachers 
had access to program 
materials & resources? 


Ea 


Fidelity of Implementation 
With respect to teaching 
points, how closely does SLIC 
teachers’ instruction follow the 
SLIC model? 


With respect to pedagogy, 
how closely does SLIC 
teachers’ instruction follow the 
SLIC model? 


How do SLIC teachers 
differentiate instruction to 
target students’ assessed 
needs? 


How do SLIC teachers make 
use of results from the 
program’s diagnostic 
assessments and student 
coursework? 


a 

i 4 
tL. 
tL i 
eae 
| ae 


Figure 5. Year 4 Data Sources on Implementation Linked with Research Questions: Targeted Intervention (continued) 
Measures/Data Sources 
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Year 1 Implementation Study 


In the first year, two high schools and three middle schools participated in the program, 
with classes led by seven teachers across the five sites. External factors led to delays in 
the program developers’ arrival in Year 1 and slowed implementation of the targeted 
SLIC program and training of coaches and teachers, a situation exacerbated at sites where 
coaches’ arrival was also delayed. Coaches provided support to teachers as they were 
acquiring basic program knowledge. To some degree, this meant that staff and student 
exposure to the full program was of shorter duration and depth than anticipated. 


However, once the coaches were in place there was strong professional development 
(PD) support for the targeted intervention teachers (at most sites) in formal PD sessions, 
site work provided by developers and district leadership, and in daily support from the 
coach. There was also variability across school sites in their exposure to PD. By the end 
of the year classroom implementation of the curriculum was fairly high®, especially at 
sites with strong teacher/coach or teacher/leadership collaboration, and the program was 
successful at implementing instruction around literacy strategies and routines such as text 
features and form. The last third, the more abstract elements of the SLIC curriculum 
such as synthesis and inference, received less attention in PD sessions or classroom 
teaching, partly due to the delays, and this became a greater priority for Year 2. 


Randomization of students into treatment and control groups was problematic in the first 
year and, as a result, only two of the five intervention schools followed the evaluator’s 
random assignments to a degree that allowed analysis. These two schools had different 
control conditions, specifically one had a Literacy Advancement Academy class for 
control students while the other school had no alternate literacy classes. 


Since there was no quantitative observation measure available in Year 1, implementation 
of the Year | classroom model was not calculated. 


Year 2 Implementation Study 


In Year 2, professional development for coaches and targeted SLIC intervention teachers 
consisted of formal sessions led by program leaders, site visits by program leaders, and 
weekly meetings of intervention coaches with program leaders. There were fewer cross- 
site sessions and more site-visits, and leadership attended more to the whole school 
program than in Year 1. Coaches’ classroom and professional development support for 
teachers also continued to be an important part of the intervention. Fidelity of 


° This statement is based on qualitative observation in SLIC classrooms: SLIC was the exclusive focus of 
instruction in SLIC classes at 4 of the 5 schools, and took 50% of the class time at the 5th school. 
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professional development implementation was estimated in the same manner as in Year 
1. 


In Year 2, fidelity of implementation inputs was lower than in Year 1, with no schools 
achieving high levels of PD fidelity. Teachers’ professional development participation 
and their direct support from coaches both declined from Year 1 to Year 2, with 88% and 
63% of sites, respectively, being rated “low.” These declines relate to substantial increase 
in the number of intervention sites (from 5 schools in Year 1 to 8 schools in Year 2) and 
intervention teachers (from 7 teachers in Year | to 20 teachers in Year 2). Leadership 
devoted more time to the whole-school program and to site-specific and department- 
specific visits along with development of the literacy assessment, activities which 
decreased the time available for professional development and coach support at the 
school site. Nonetheless, the rating for coaches’ professional development increased 
somewhat in Year 2. 


Ratings of classroom implementation were based on measures of five key program 
elements. These are the use of grade-level texts, coverage of the program curriculum, 
assessment of student needs and differentiation of instruction, attention to metacognitive 
skills, and scaffolding to student independence. All schools were rated as having a 
“medium” level of fidelity of implementation. However, underlying this rating was 
considerable variation in fidelity across classrooms within a school, and variation among 
components of the rating criteria as some components (such as independent work and 
whole-class metacognitive questioning) were negatively correlated or reflected different 
instructional strengths. Variation across sites in professional development training and 
support does not seem to be connected to variation in fidelity of classroom instruction. 
This might be due to the connection of training and support to varying levels of teacher 
need. 


Fidelity of implementation was slightly higher, on average, in intervention high schools 
than in middle schools. In middle schools, intervention teachers’ exposure to professional 
development ranged from low to medium, while high school ratings were distributed 
between high, medium, and low. Middle and high schools both fell in the range of 
“medium” level of ratings for fidelity to the SLIC instructional model, however, the 
average high school rating (2.13 on a 3-point scale) is higher than the middle school 
rating (1.66). This pattern remained small but consistent for each subcategory of the 
ratings, metacognition, assessment/differentiation, coverage of the curriculum, 
scaffolding to independence, and use of grade level texts, where the high school ratings 
were higher by .08, .3, .44, .42, and 1.13 respectively. 


Year 3 Implementation Study 
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In Year 3, the SLIC program developers/leadership made substantial changes to 
professional development delivery and provided new curriculum materials to targeted 
SLIC teachers at intervention schools. Coaches were directed to turn their attention to 
promoting the whole school program, while the leadership worked more directly, 
individually, and frequently with intervention teachers. The result was a shift in coaching 
responsibilities from coaches to developers and, arguably, more “on model” guidance for 
teachers. In addition, the proportion of experienced SLIC teachers was higher than in 
prior years, which meant less need for PD at most sites. At the same time, there was 
greater support for new teachers, and a significant increase in material scaffolds. These 
factors may have increased the quality if not necessarily the amount of PD provided to 
intervention teachers. 


In Year 3, schools achieved a medium level of fidelity to the classroom model. In Year 3, 
fidelity to the classroom model was estimated based on observation of classes, surveys, 
and interviews with teachers, coaches, and program leaders. Scores were calculated for 
five different components of the classroom model (use of grade level texts, scaffolding to 
independence, coverage of the program curriculum, assessment of student 
needs/differentiation of instruction, and metacognition), which were then weighted by the 
number of classes the teacher taught and summed together. 


With respect to the core program goal of differentiation of instruction there was some 
growth seen in a student survey question about SLIC teachers’ provision of written 
feedback on work, from 2.05 on a 4-point scale to 2.35, with three schools showing 
progress in Year 3. Other components of fidelity, such as use of grade level texts and 
scaffolding to independence, showed no growth. Interviews conducted with leadership at 
the end of Year 3 indicate they were aware of ongoing challenges in implementation, and 
had thought about how to address them. 


Year 4 Implementation Study 


Year 4 saw further changes in the delivery of professional development and increases in 
aspects of instructional fidelity at some sites but no overall gains across sites and 
measures. Leadership, in contrast, viewed the veteran SLIC teachers and the majority of 
sites as having a high level of fidelity with medium fidelity among new teachers and at 
schools with high turnover of personnel over multiple years. 


There was a Year 4 decrease from Years 1, 2, and 3 in the proportion of schools receiving 
the amounts of professional development support defined as adequate by developers in 
Year 1. (See Tables 1 and 2 for professional development summaries across years.) The 
type and amount of professional development was less consistent and, in some respects, 
was increasingly focused on the needs of teachers and sites, continuing a trend towards 
differentiation of support that began in the second year. Coach meetings were no longer 
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held, and the cross-site work diminished to one event held over a few days before the 
school year began and assessment scoring sessions during and after the school year. The 
common program resources and cross-site linkages, aside from classroom materials and 
an additional preparation period for targeted intervention teachers, consisted of the 
professional development time spent with developers and the district leader and the 
adaptable curriculum units developed prior to Year 3 and revised before Year 4. Four 
schools had site-based coaches, in contrast to eight schools in Year 2, and their 
interaction with developers tended to be on-site rather than at program-wide events. The 
curriculum units and the time spent with developers were generally praised by teachers as 
supportive of classroom implementation, and this was especially the case among new 
teachers. With the support of leaders and curriculum units, the new teachers moved 
quickly towards competence and are expected to be fully proficient by the end of their 
second year (Year 5). 


The leadership spent most of their time with new teachers, while veteran teachers were 
considered able to proceed on their own. Some veteran SLIC teachers were satisfied with 
the level of support they received, but a couple of teachers commented that they had lost 
some of the advantages of working with a professional community focused on literacy 
instruction, such as the shared reflection on student work, exchange of ideas, and 
observation of their instruction. While professional development was low at half the 
sites, it may have been adequate to maintain the targeted intervention at those sites given 
the competence of veteran SLIC teachers. 


Change in personnel represents a challenge for a program based on the acquisition of 
instructional skill and knowledge, although the curriculum units made the process much 
smoother and faster in Year 4 than it had been in Year 2. Two of eight school sites saw 
no turnover in SLIC teachers, and one of those sites also kept its coach and principal. 
There were new principals at two sites, and one of the original SLIC leaders left the 
program. Five of eight coaches left the program at the end of Year 3 as did four of 
sixteen teachers. A SLIC teacher who had been with the program since Year 1 moved 
into the role of coach at one school, and another coach served two schools, as she had in 
Year 3. Other SLIC teachers provided Whole School professional development at their 
sites. Thus there were four coaches serving five of the eight schools, and they were 
tasked with furthering the content area Whole School program implementation, while the 
leadership continued their Year 3 focus on SLIC teachers. Five of the sixteen Year 4 
teachers were new to the program, two teachers were on maternity leave for half the year 
and the coaches stepped in to lead the SLIC classes, and another teacher left mid-year and 
was replaced. It is possible that low-performing school sites have higher personnel 
turnover as well as a high proportion of teachers new to teaching. 


By Year 4, the changes in the organization of professional development support were so 
substantial that the distinctions used in earlier reports (support from leadership, direct and 
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indirect support through coaches) no longer made sense, so only the overall measure of 
support is represented. 


Table 1 presents the Average Professional Development Adequacy Score in middle and 
high schools in Years | through 4. In Year 4, half of the high schools and half of the 
middle schools had low ratings on the Professional Development Adequacy Score. The 
other half of the middle schools reached medium adequacy, while one high school had 
medium and another had high levels of professional development. This represented a 
general decrease in professional development from prior years. 


Table 1. Average Professional Development Adequacy Score: Level of 
Implementation of the Targeted SLIC PD Model, by School, Years 1-4 


Fidelity Middle Schools High Schools 
Year 1 
Low 0 0 
Medium 67% (2) 50% (1) 
High 33% (1) 50% (1) 
Year 2 
Pow 50% (2) 0 
Medium 50% (2) 100% (4) 
Year 3 
Low 0 0 
Medium 100% (4) 100% (4) 
Year 4 
Low 50% (2) 50% (2) 
Medium 50% (2) 25%(1) 
High 25% (1) 


Note. Percentages based on 5 schools in Year 1, and 8 schools in Years 2-4. 
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Year 4 Classroom Implementation 


The Average Classroom Adequacy Score changed little from Year 3 to Year 4, and all 
schools achieved medium level of fidelity of classroom implementation in Year 4. 
Fidelity is assessed on five factors, with data derived from student and teacher surveys, 
interviews, classroom observations, and reading materials collected during observations. 
The five factors used to assess fidelity are 1) scaffolding to student independence; 2) use 
of grade level texts; 3) coverage of the SLIC curriculum; 4) assessment of student work 
and differentiated instruction; and 5) metacognition. The Year 4 ratings for metacognition 
are low, and only two sites reached medium fidelity on this factor. Ratings for the use of 
grade level texts are high, and ratings of the other factors are medium. There was 
virtually no difference between the scores of middle and high schools. As noted above, 
the leadership had a different view of the level of implementation achieved, and viewed 
most sites and virtually all continuing SLIC teachers as having high fidelity of 
implementation in their classrooms with new instructors achieving medium fidelity. 

(See Table 2.) 
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Table 2. Level of Fidelity to the Targeted SLIC Classroom Model, Years 1-4, by 
School 


Average Classroom Adequacy Score 


Year | Year 2 Year 3 Year 4 


Middle Schools 


Low -- 25% (1) 
(<1.5) 


Medium -- 75% (3) 100% (4) 100% (4) 
(1.5-2.5) 


High -- 


(>2.5) 


High Schools 


Low -- 


(<1.5) 


Medium -- 100% (4) 100% (4) 100% (4) 
(1.5-2.5) 

High -- 

(>2.5) 


Note. Percentages based on 5 schools in Year 1, and 8 schools in Years 2-4. 


Table 3 displays the implementation fidelity scores for the four years of the program, 
including the Average PD Adequacy Score in Year 1, when no classroom measure was 
available, and the average of PD Adequacy Scores and Classroom Adequacy Scores in 
Years 2 and 3. This, in effect, presents the overall implementation fidelity over four 
years and shows growth from Year 2 to Year 3, from 75% reaching medium fidelity to 
100% achieving medium fidelity, and consistency between Year 3 and Year 4. 
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Table 3. Level of Implementation of Targeted SLIC PD and Classroom Models, 
Year 1-4 


Average PD Adequacy Average PD and Classroom Adequacy Score 


Score* 

Year 1 Year 2 Year 3 Year 4 
Low 0 25% (2) 
(<1.5) 
Medium 60% (3) 75% (6) 100% (8) 100% (8) 
(1.5-2.5) 
High 40% (2) 0 
(>2.5) 


Note. Year 1 scores did not include the classroom model. In Years 2 and 3, the Average 
Adequacy Score is the average of teacher PD participation, average coach support, and 
classroom fidelity. Percentages based on 5 schools in Year 1, and 8 schools in Years 2-4. 


Components Contributing to Classroom Fidelity 


Assessment of Student Work and Differentiated Instruction. The mean fidelity rating for 
differentiation decreased from 2.17 in Year 2 to 1.54 in Year 3 and increased to 2.37 in 
Year 4. Four schools had high fidelity on this variable and two had medium ratings. 
From Year 3 to Year 4 there was an increase in ratings at four schools, a decrease at 
three, and the same rating another school. In interviews, teachers were much more able 
to articulate the way they went about differentiating instruction and seemed more 
confident in their skill. While prior years had seen substantial amounts of time spent on 
developing the assessments, in Year 4 assessments were used principally as professional 
development tools and additionally to provide information on student progress. Teacher 
surveys suggested that review of student work provided the most useful information on 
student progress. 


A question on the student survey asks how often teachers provide written feedback or 
comments on their work, and asks the question for each content area and for SLIC. (See 
Table 4.) Response options include “not in this class” (0) and range from “never” (1) to 
“every day” (4). First, there is a difference between SLIC assigned students and students 
who were both assigned and enrolled, and both Year 3 and Year 4 saw higher (more 
positive) response among those who were assigned and enrolled than those who were 
assigned but not enrolled in SLIC classes. Second, while there is no increase across years 
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among those assigned but not enrolled, there is a jump in response from Year 2 to Year 4 
for those assigned and enrolled, with somewhat lower ratings in Year 3. High school 
students gave more positive responses than did middle school students. There was 
considerable range of ratings associated with SLIC classrooms but much less variation 
across schools. The response to the parallel question for English Language Arts classes 
is offered to provide perspective. 


Table 4. Teachers' Provision of Written Feedback on Student Work in Literacy 
Advancement Class, by Program Year 


Year 2 post Year 3 Year 4 
Mean SD Mean SD Mean SD 
SLIC Assigned 2.45 0.92 2.33 0.95 2.38 1.00 
SLIC Assigned and Enrolled 2.41 0.90 2.37 0.94 2.69 0.89 
- Middle School SLIC 
(Assigned, Enrolled) 2.38 0.89 2.34 0.93 2.66 0.76 
- High School SLIC 
(Assigned, Enrolled) 2.44 0.91 2.43 0.96 2.71 0.97 
Intervention School ELA Feedback 
(SLIC Assigned Students) 2.37 0.80 2.28 0.80 2.54 0.80 
Control Students ELA Feedback 2.44 0.72 2.30 0.77 2.45 0.75 
Intervention School ELA Feedback 
(all students) 2.43 0.77 2.44 0.78 2.69 0.91 


Note. Student survey question (4e), “How often do your teachers usually give you written 
feedback or comments on your work ... in extra class for reading and writing (e.g., 
SLIC)?” Response options include “not in this class” (0) and range from “never” (1) to 
“every day” (4). 


Assessment of student progress is another piece of the differentiation-assessment rating. 
According to teacher response to survey questions (how many times did you use SLIC 
assessments to inform your instruction; use mandated assessments to inform your 
instruction; review student work as formative assessment) over the three years there was 
an increase in the attention SLIC teachers have given to student work in particular. In 
interviews, a few teachers observed that assessment and differentiation take time, and that 
it would be difficult to carry out these aspects of the program model without support of a 
coach or leadership and with a full complement of students and no extra preparation 
period. 
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Grade Level Texts. According to the program model, students should be assigned grade 
level texts and readings should not consist of low-level materials. Texts were collected 
during classroom observations, and measured using the Flesch-Kincaid Grade Level Test. 
If we look at the degree to which texts diverge from grade level, the divergence is lowest 
in Year 2, and this the way fidelity scores have been calculated. Considered another way, 
the results are somewhat different. Across the eight schools, a higher proportion of 
reading materials were at or above grade level (61%) than below grade level (39%) in 
Year 4. The reverse was true in Years 2 and 3, with 46% at or above and 53% below in 
Year 2, and 44% at or above in and 56% below grade level in Year 3. In the middle 
schools, the ratings dipped from Year 2 to Year 3, and were highest in Year 4. In high 
schools, ratings declined slightly from Year 2 to Year 4. (See Table 5.) 


Table 5. Average Grade Level of SLIC Texts, by Program Year 


Grade Level(s) Average Grade Level 
Year 2 Year 3 Year 4 

a 6.76 7.07 8.35 
7"/8th 12.92 5.66 6.63 

8 8.84 6.99 6.68 

9 8.82 9.63 8.83 
9""/10th 10.39 6.83 8.16 
10 9.98 9.99 11.11 


Note. Grade level determined using Flesch-Kincaid Grade Level Readability Test. 


Metacognition. Ratings for metacognition were slightly lower in Year 4 than in prior 
years, and only one school showed increased instruction in metacognition. The average 
middle school rating was low, and the average high school rating was medium, and there 
was a point and a half difference (on a three-point scale) between the lowest and highest 
ratings across classrooms. Over the years, the highest ratings were in Year 3. While the 
rating is reasonably reliable within a given year, it is not reliable over time because it 
based exclusively on classroom observations, and the measurement and calculation of 
metacognition changed from Years 2 through 4. Leadership did not express a common 
view of Year 4 progress on metacognition, and some indicated that there was a strong 
showing among veteran SLIC teachers while others observed that experienced SLIC 
teachers were finding multiple ways to encourage students to reflect on their work, but 
they saw little evidence of student reflection on their learning. Teachers stated that they 
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used questioning and entrance and exit slips and some used journals to encourage 
metacognition, but they did not expect that students were using metacognitive approaches 
independently. 


Coverage of the Curriculum. Four schools had high ratings for coverage of the 
curriculum in Year 4, and the other four schools received medium ratings. The average 
rating across schools changed from 2.05 in Year 2 to 2.2 in Year 3 and 2.39 in Year 4. 
Six schools had higher ratings in Year 4 than in Year 3, while ratings declined in two 
other schools. 


A set of questions on the student survey provides additional information on coverage of 
the curriculum. Students were asked how often they read in their literacy advancement 
(SLIC) class, and how often the write, preview text features, underline important details, 
take notes, and use notes to help with an assignment. The response options were “0” (not 
in this class”) and ranged from “never” (1) to “always” (5). To combine the question set, 
the “zero” responses were removed. On these questions, between five and eight percent 
of those assigned to and enrolled in SLIC classes responded that they were not in the 
class. These responses appeared across the schools, but were mostly clustered in schools 
that used other class titles for SLIC. There was an increase in coverage of the SLIC 
curriculum noted by students in those classes. There was little difference in response 
between middle and high school students. (See Table 6.) 


Table 6. Mean Literacy Instruction in SLIC Class, by Program Year 


Year 2 post Year 3 Year 4 

Mean SD Mean SD Mean SD 
SLIC Assigned 4.16 1.11 4.19 1.07 4.25 1.12 
SLIC Assigned and Enrolled 4.12 0.95 4.25 1.03 4.59 0.66 
- Middle School 4.12 1.14 4.27 1.01 4.58 0.56 
- High School 4.07 1.29 4.24 1.06 4.59 0.66 


Note. Student survey questions (10a-f), “How often do you do the following in your 
literacy advancement class (e.g., SLIC)....read in class/write in class/preview text 
features/ underline important details/take notes.” Response options include “not in this 
class” (0) and range from “never” (1) to “always” (5). 


Scaffolding to Independence. SLIC teachers are expected to assign students independent 
work, particularly in reading and writing, and to provide scaffolding but allow students to 
read and write for increasing periods of time on their own. Fidelity ratings were based on 
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classroom observations of both independent work and scaffolding during independent 
work periods, and this portion of the protocol remained fairly consistent over the course 
of the evaluation. Fidelity ratings on this variable remained fairly stable over the years, 
with a rating of 1.688 in Year 2 to 1.625 in Year 3 to 1.575 in Year 4. Ratings were 
higher in middle schools in Year 3, and in high schools in Year 2 and 4. Two middle 
schools and a high school had higher ratings in Year 4 than in prior years. 


Students’ Reading Practices 


A set of questions on the student survey enquires into students’ use of literacy practices 
advocated by SLIC, particularly for reading expository text. In the view of SLIC 
developers, these practices are characteristic of proficient readers. The response options 
run from zero (“not true for me’) to 4 (“very true for me”) and five of the seven questions 
are included in this set, while the individual questions are presented below. The two 
items excluded from the summary set either contradict SLIC practice or ask a different 
sort of question. The practices advocated by SLIC are “when I become confused about 
something I am reading, I go back and try to figure it out,” “before I study new material 
thoroughly, I often skim it to see how it is organized,” “when studying, I try to figure out 
which concepts I do not understand,” “I think the best way to read a textbook chapter is 
to go back and forth in the chapter to check my understanding,” and “I have different 
ways of reading fiction and non-fiction.” Another question is not advocated by SLIC and 
is not included in the aggregated means, namely “I think the best way to read a textbook 
chapter is to read from beginning to end without stopping along the way.” Another 
question is somewhat less relevant, and asks about confidence in reading textbooks, “if I 
read a textbook, it will make sense to me.” 


It should be emphasized that this is not a longitudinal analysis. However, since there 
were no new random assignments in Year 4, most of the control students and the 
randomly assigned treatment students are the same students in Year 3 and Year 4. The 
“assigned and enrolled” students in Year 4 are generally those students who did not test 
out by the end of Year 3, or who had tested out in Year 2 or 3 and tested back in for Year 
4, 


Table 7 provides summary ratings for the set of questions reflecting practices advocated 
by SLIC. The first thing to note is that the mean responses are not very high. Second, 
there very is little difference between randomly assigned treatment students, control 
students, and Year 4 assigned and enrolled students. Fourth, there is a slight increase in 
response for both treatment and control students from Year 3 to Year 4. Fifth, SLIC may 
have had more influence on some reading habits than on others. SLIC instruction may 
have increased the practices of previewing and cross checking (11b and 11f) and may 
have made less difference on reviewing (11a), or understanding differences between 
reading fiction and non-fiction (11g). It must be repeated that differences are small. 
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Table 7. Summary of Students’ Reading Practices 


Year 2 Year 3 Year 4 
Mean SD Mean SD Mean SD 
SLIC Assigned Students 2.45 0.90 2.38 0.96 2.54 0.91 
SLIC Assigned and Enrolled 2.41 0.89 2.34 0.97 2.46 0.94 
- Middle School 2.40 0.91 2.37 0.97 2.41 0.87 
- High School 2.41 0.89 2.28 0.96 2.49 0.98 
Control Students 2.43 1.00 2.32 0.94 2.51 0.87 


Note. Aggregate of selected student survey questions: “When I become confused about 
something, I go back and try to figure it out.” (11a); “Before I study new material thoroughly, I 
often skim it to see how it is organized.” (11b); “When studying, I try to figure out which 
concepts I do not understand well.” (11c); “I think the best way to read a textbook chapter is to go 
back and forth in the chapter to check my understanding.” (11f); and “I have different ways of 
reading fiction and non-fiction.” (11g). Response options range from “not true for me” (0) to 
“very true for me” (4). Reliability analysis shows an alpha of .676 for the Year 4 survey. 


Variability in Implementation of Targeted SLIC 


In Year 4, there was somewhat less difference in fidelity of classroom implementation 
among teachers at each site than there had been in previous years. Intra-school 
implementation differences in Year 2 were often associated with the presence at the same 
school of experienced SLIC teachers and teachers new to SLIC and to teaching. In Year 
3 differences among teachers at some schools seemed related to the length of their tenure 
in SLIC, their eventual decisions about whether to remain with the program, general 
teaching skills and rapport with students. By Year 4, the curriculum units had addressed 
some of the differences associated with tenure in SLIC, and the new teachers in Year 4 
were not new to teaching. The Year 4 differences among teachers were sometimes 
associated with their rapport with students, their ease with some of the higher level skills 
such as critical reading, and what one of the developers referred to as whether they had 
“made it [SLIC] their own.” 


Year 4 differences among classrooms were apparent in the student surveys. Across all 
teachers in Year 4, there was a mean 1.0 difference among teachers on a five-point scale 
regarding coverage of curriculum in their SLIC class, and a 1.0 difference in response to 
a question about provision of written feedback. There was a smaller difference in 
response, 0.8, to questions about students’ use of literacy practices advocated by SLIC. 
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In Year 3, student survey responses ranged from 0.2 and 0.46 (on a 5-point scale) 
between grade levels within schools (and a maximum difference of 0.8 between schools), 
while interviews with coaches, leadership, and teachers pointed to far greater differences 
among teachers, particularly in instruction in the more abstract reading skills. Intra- 
school differences were greater on the differentiation of instruction criteria, and at four 
schools there was a difference of 0.5 to 0.7 (on a 4-point scale) in student survey response 
regarding the frequency with which teachers provide feedback on their work (and a 
maximum difference among schools of 0.64). There were substantial differences among 
teachers within schools on the metacognition rating, and four schools showed more than 
0.5 difference (on a 3-point scale) between teachers, and one with a 0.85 difference and 
another with a 0.97 difference. Teachers were generally strong or relatively 
inexperienced with respect to more than one of the SLIC fidelity criteria. Thus the 
medium levels of school implementation in some cases represent high and low 
implementation in different classes at a school rather than a medium level of fidelity in 
all classes. 


Middle School / High School Differences in Fidelity of Implementation 


A comparison of classroom fidelity of implementation scores for middle and high school 
suggests more growth in middle school scores from Year 2 to Year 3 than in high 
schools, and slightly higher scores in high schools than in middle schools. The high 
schools scored very slightly higher (0.1) in students’ response to a 4-point survey 
question about the amount of feedback they receive in SLIC classes, which relates to the 
goal of differentiated instruction. However, student survey response to a series of 
questions about coverage of the curriculum among middle and high school students was 
virtually the same. The more substantial difference between middle and high schools was 
in organizational variables such as higher turnover of teachers, coaches, and principals at 
the middle schools and the presence in Year 3 of an onsite coach who continued to 
support SLIC teachers. In Year 4 there was more growth in fidelity scores at middle 
schools than high schools, and less difference between middle and high school average 
fidelity scores than in Year 3. The overall middle/high school difference was negligible. 
There was less stability in coaching staff at high schools than the middle schools between 
Year 3 and Year 4. Prior years had seen greater stability and stronger implementation at 
high schools 


Program Changes Years 1-4 


This section summarizes some of the changes that occurred at targeted intervention 
schools and within the program between fall of 2005 and spring of 2010. 


Professional Development. There were changes in professional development delivery 
from the first to the fourth year of the program. The program strongly advocates the value 
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of teacher professional development and teachers’ agency in classroom instruction, while 
the program model does not designate a single formula for PD delivery but, rather, 
proposes that PD delivery should adapt to school context and teacher need. The delivery 
of PD in the first year involved large cross-site events for teachers and coaches and 
weekly coach meetings. In turn, coaches were expected to support and inform teachers. 


In the second year leadership shifted to more site-based work and there were far fewer 
cross-site events, while the model of leadership supporting coaches who supported 
teachers remained intact. In addition, the leadership spent considerable time on content 
area work and assessment development. 


The leadership began to question the efficacy of this effort and noticed the strengths of 
particular coaches, and in the third year they organized a team of coaches to provide PD 
to content area teachers while leaders and developers supported intervention teachers 
directly and provided curriculum units for each grade level. Since the coaches at two of 
the sites had been frequently unavailable in Year 2, leadership may have provided a net 
increase in support in Year 3. The frequency of coach meetings also decreased in Year 3, 
and they ended entirely in Year 4. At the same time, the leadership came to favor the 
idea that, in any future implementation of SLIC, coaches might start out as teachers in the 
program and should have experience enacting the instructional model before coaching 
others in its implementation. This new approach took shape in Year 4 as one experienced 
SLIC teachers moved into a coaching role, and a couple of teachers at other sites offered 
professional development for content teachers. The leadership focused their efforts on 
SLIC teachers and classes, particularly new SLIC teachers, while the remaining coaches 
worked in the content areas. 


The cross site team of middle school content coaches created in Year 3 did not continue 
in Year 4. In Year 4, the four coaches who remained were focused mainly on content area 
professional development at their school sites, and one coach worked at two schools. One 
of the middle schools and the two largest high schools did not have a coach in Year 4. 
Coaches provided some support to new teachers at their sites, but it was expected that 
continuing SLIC teachers would generally not need support for SLIC classes, though they 
were supported in offering content area professional development at their sites. The 
result of this was a very mixed pattern, of high levels of support at some schools and 
virtually no support at others. Teachers who had strong professional communities at their 
school sites adapted well to this, while others believed or hoped that it was a sign of the 
leadership’s confidence in their practice, and some regretted the loss of a professional 
community. 


As in prior years, one developer provided support to three middle schools and one high 
school, while the other supported one middle school and three high schools. The 
developers have different styles in interacting with coaches and teachers and they may 
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place different emphases on aspects of program pedagogy’, while their approach to 
literacy is closely aligned. 


Schools and Personnel. The two developers, one of two district leaders and one principal 
remained with the program for four years of the grant and are continuing in their fifth 
year. Only one person from the ranks of teachers and coaches remained all four years. 
The three schools that joined the program in Year 2 have retained their principals, two 
have retained their coaches, and five of eight original SLIC teachers at those schools 
remain. 


In a program based in professional development and relying on trained and skilled 
teachers, teacher turnover has presented a challenge for leadership. Turnover of personnel 
was much lower in Year 3 than in the second year of the program, and the majority of 
Year 3 SLIC teachers had been with the program the prior year, while one teacher and 
three coaches remained from Year |. In Year 3, only three of sixteen teachers, 19%, were 
new and one of these was a coach and thus very familiar with the program. This contrasts 
with Year 2, when 89% of the teachers were new to the program.® At the end of Year 3, a 
quarter of the teachers left, and about a third of the Year 4 teachers were new. Coach 
turnover was higher, and about half the coaches left at the end of Year 3, leaving three 
coaches in Year 4; one coach was replaced by an experienced SLIC teacher. 


There was decreasing rate of departure of SLIC teaching personnel from the schools from 
Year 2 to Year 4, although the proportion of new SLIC teachers was fairly consistent 
between Years 3 and 4. By Year 4, there were no teachers or coaches remaining from 
Year 1, although one coach was replaced with a Year 1 teacher. Turnover of coaches 
and teachers has been reflective of issues at the site and, far more frequently, within the 
program. However, in exit interviews with teachers and coaches, most praised the 
instructional model and, when asked which aspects they would take with them, many said 
“all of it” although some would use other forms of instruction as well. (See Table 8.) 


7 Based on interview data for example, it seems that one may place somewhat more emphasis on student 
data and the other on questioning strategies. 

* Five of eight teachers (63%) left after the first year, as did two of five (40%) coaches. At the end of the 
second year, six of nineteen (32%) of Year 2 teachers left, some because of a school closing, and all 
coaches remained and seven of the 8 were active in Year 3. 
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Table 8. School and Personnel Changes in Targeted SLIC Intervention, Years 1-4 


Year | Year 2 Year 3 Year 4 
Schools 1 HS school 3 schools added 1 school closed No change. 
left program (1 MS, 1 small HS for failure to meet 
early in Y1 & 1 AYP; new school 
comprehensive at same site, with 
HS) to increase new principal, 
sample size staff, and some 
(not all) SLIC 
students from 
prior school 
Teachers 63% Yi teachers 32% Y2teachers 25% Y3 teachers 
left; 89% Y2 left; 25% Y3 (4 of 16) left; 
teachers new teachers new 
31% of Y4 
teachers (5 of 16) 
new in fall; 1 new 
in 2™ semester 
(total 38%); 2 
others on 
maternity leave 
part of Y4 
No teacher 
change at 25% (2 
of 8) schools; 
Coaches 40% Y1 coaches One coach 62.5% (5 of 8, 2 


left; 66% Y2 
coaches new due 
to addition of 
schools; 12% of 
Y2 coaches on 
leave 


assigned to non- 
coaching duties 


1 coach assigned 
two schools 


MS and 3 HS) 
coaches left, 
including the one 
assigned non- 
coaching duties in 
Y3. 


1 coach continued 
at two middle 
schools. 


1 new HS coach, 
former SLIC 
teacher 
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Year 1 Year 2 Year 3 Year 4 


Leadership One of leaders 


assigned other One leader left 


(assigned other 


duties 
duties in Y3) 
Principals 20% (1) left 12% (1) Y1 25% (2) Y2 25% (2 of 8) Y3 
school at start interim principal principals left; principals left; 
of year; empty left; 50% (4) Y2 25% Y3 25% (2 of 8) Y4 
post, interim principals new principals new principals new 
hired 
District Empty post Superintendent New (Y2) Interim 
superintendents _—_ until end of named at end of superintendent superintendent 
academic Y1 Y1 left, another left in Y3; Interim named as 
hired in Y2 named superintendent 


A major example of large-scale ‘turnover’ came when one of the original intervention 
middle schools closed at the end of Year 2 after failing to meet AYP for five years, and a 
nearby school transferred to the site of the intervention school and opened in Year 3 with 
a new principal and many new staff members and students along with some continuing 
students from the old school. The school closing had less impact on implementation of 
the targeted intervention than on the whole school program because one of two 
intervention teachers at the new school was an experienced SLIC teacher. Change at this 
school continued as the principal and both SLIC teachers left at the end of Year 3 and 
were replaced with new personnel in Year 4, and these changes affected both the targeted 
and the whole school program. The coaching position also changed across the years, and 
there was little consistency at this school across any two years. 


The position of superintendent was more stable from the end of Year 3 through Year 4, 
although the new superintendent did redesign the district’s organizational chart and the 
way content areas, teachers, and schools were supported, as had his predecessors. Since 
the arrival of a new superintendent is often associated with introduction of new policies, 
departmental reorganization, and new principal assignments, a change at the top has 
practical implications for school sites and contributes to a general sense of uncertainty. 
Principals’ support for the program continued to vary by school, and the remaining 
coaches were most responsible for the interface between the program and the schools. 


The demographics and character of one of the small schools may have changed in Year 4 
due to economic influences. The coach and principal mentioned that the economic 
downturn in California had made it more difficult for some low income families in the 
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neighborhood to afford private (largely religious) schools and more of those students 
were now attending the public school. As a result, the student population and academic 
focus of the school had shifted somewhat as more students sought out advanced 
placement (AP) classes and college preparatory courses. 


Program Developments and Adaptations 


To increase implementation and respond to the ongoing change in the schools, a variety 
of program developments were undertaken. (See Table 9). In Year 3, the leadership 
provided teachers with a set of grade-level curriculum units written during the summer 
before Year 3 with the participation of some teachers and coaches. The set included an 
overview unit, a unit on expository text, another on persuasive text, a fourth unit on 
narrative text and a research unit. Each unit contained a matrix outlining the lessons 
along with the skills and strategies taught and classroom activities associated with each. 
The units contained lesson plans and reading materials that teachers can use or replace 
with other readings. In addition, the overview unit provided the foundational documents 
of the program such as the Scope and Sequence, and research papers representing the 
program’s pedagogical approaches. 


Response from coaches and teachers to the materials was very positive, and their 
comments indicated that the inclusion of reading materials for students was very helpful 
and saved them time, and that the lessons provided a more concrete plan for the year 
although bringing it to life was still left to them. Leaders expressed satisfaction and some 
concern that the availability of such a plan might discourage teachers from creating 
lessons responsive to the daily needs of their students. There may have been variation in 
the use of the units across sites, and the majority the texts collected during classroom 
observations did not come from the units. 


The provision of grade-specific curriculum allowed some progression of instruction for 
students who remained in the program. The progression was not based on whether the 
student had been in the program for one, two, or three years (their years of exposure), but 
rather on the grade level of the intervention class. There was concern in the Ds year that 
students might be presented with the same texts in multiple grade levels and that student 
engagement would fall from levels that some teachers considered already low. The 
progression of skills and classroom activities outlined in the units is similar across grades, 
while the reading materials are different and the lesson plans are somewhat different 
across grade levels. Reading materials were chosen for their value in illustrating and 
advancing instruction of the teaching points of each lesson, with attention to the 
complexity of texts to address the goal of providing students with grade level reading 
materials. 
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Since students are exposed to the same concepts and activities and are instructed in the 
same skills and strategies across years, students can receive full exposure to the program 
in a year. However, an increase in implementation, additional practice, and fuller 
articulation of the program from Years | to 4 may make a difference for those entering 
later or remaining longer. 


In an effort to increase student engagement, more magazine articles and narrative texts 
were used in Year 3. This contrasts most strongly with Year 1, when textbooks were the 
most common source of expository reading materials. In Year 4 there was some use of 
textbook materials in SLIC classes, but the collection of materials during observations 
suggests that expository text consisted primarily of magazine articles. In interviews, 
teachers noted that finding interesting topics was one of the most significant factors 
affecting student engagement, and they worked to provide materials that students would 
want to read. 


In Year 3, leadership asked teachers to collect student data and work with “focus 
students” as a form of differentiated instruction. The emphasis on “focus students” 
diminished in Year 4. The interest in analyzing student work as a way to guide 
differentiated instruction has been a central element of the program since the first year 
and, in Year 3, the possibility of collecting student data was expanded by a web-based 
application purchased by the District which allowed tracking of student performance on 
statewide tests by teachers and coaches. The leadership worked closely with teachers in 
analyzing student work, and this continued in the fourth year. 


There were some shifts in the ideas about small group work and related changes in 
classroom practice. In Year 1, emphasis was placed on independent work as a way to 
give students more experience working with texts independently as part of the goal of 
increasing literacy independence in general. While the goal of increasing student 
independence remained a core element in the program, in the third and fourth year the 
leadership encouraged small group work in an effort to contribute to the creation of 
academic community and increase student engagement. 


During the first and second year, a number of teachers and coaches believed that 
differentiated instruction would be carried out by separating students into groups with 
similar needs and providing differentiated instruction. The conception of differentiation 
expanded over the following two years, and in the third year teachers were writing 
comments on student work and in some cases holding individual conferences with 
students to discuss learning goals. Teachers developed a more flexible interpretation of 
how to approach differentiation and, with greater confidence and experience, were 
somewhat more willing to attempt it. 


a7 


Teachers and coaches praised the diagnostic assessment rubric as an extremely useful and 
powerful professional development tool and as representing a good alignment of the 
skills assessed and those taught. However, a few questioned its utility for assessment of 
student progress, arguing that the prompts do not explicitly state the depth and extent of 
answers students are expected to give to receive good scores. 
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Table 9. Targeted SLIC Program Changes, Years 1-4 


Year | Year 2 Year 3 Year 4 

2006-07 2007-08 2008-09 2009-10 
Participants 
# of middle schools 3 4 4 4 
# of high schools a2 4 4 4 
# of new principals 5 4 1 2 
Change in x x x x 
superintendent 
# of SLIC teachers, 8 (7 at any one 19 16 16 teachers 
total time; mid-year at any time; 

change) 18 total 
# of coaches 5 8 (one on leave) 8 (one 4 

given non- 
coaching 
duties) 

PD delivery 
More site-based work, x x x 
fewer cross-site events 
Coach becomes x x 
teacher; coach starts as 
teacher; Teachers given 
W-S PD 
responsibilities 
Leaders/ developers x x (diminished) (rarer) 
supported coaches who 
supported teachers 
Leaders/ developers X (primary x (primary 
worked directly with source of source of 
teachers PD) PD) 
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Year 1 Year 2 Year 3 Year 4 
2006-07 2007-08 2008-09 2009-10 
Teachers’ PD x x (rare at (at 4 
some schools with 
schools, coach; 
none at absent at 4 
others) schools) 
Cross site events for (throughout year) (summer, & (summer (summer 
SLIC teachers fewer only) only) 
throughout year) 
Leaders’ individualized x x (more than x (most x (for new 
work with teachers Y1) common teachers; 
form of PD) rare for 
others) 
Coaches PD 
Weekly coach x x (rare) (discontinued) 
meetings led by 
leaders 
Materials 
SLIC assessments & X (in initial x (in final x X 
scoring guides development) development) 
Foundational x x x x 
document: Scope & 
Sequence 
New documents: x x (continued 
Progression of use; no new 
Instruction, “SLIC V” documents) 
Curriculum units x x (used, 


written by developers; 
significant increase in 
material scaffolds 


modified with 


additional 
readings) 
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Year 1 Year 2 Year 3 Year 4 
2006-07 2007-08 2008-09 2009-10 
Text focus Expository Overview, Written Written 
expository, curriculum curriculum 
persuasive, units include —_ units include 
narrative, research. optional optional 
sources. Units sources. Units 
are overview, are overview, 
expository, expository, 
persuasive, persuasive, 
natrative, narrative, 
research. research. 
Reading sources textbooks high interest high interest _ high interest 
articles in teen articles in articles, 
magazines (more teen novels, some 
than textbooks), magazines textbook 
short stories, (more than 
novels textbooks), 
short stories. 
novels 
Program emphasis 
Leadership focus Targeted and Whole school Targeted Targeted 
whole school intervention intervention intervention 
interventions 
Coach focus Targeted and Targeted Whole school Whole school 
whole school intervention intervention intervention 
interventions (less on (less on 
targeted) targeted) 


Changes in Implementation from Year 1 to Year 4 


Over the course of the four years there was a substantial increase in the number and depth 
of material scaffolds for teachers such as documents and curriculum units, with the most 
marked change and provision of the most practical materials appearing in Year 3. The 
substance of the program became much more clearly and fully articulated over the course 


of four years, and the number of supports increased. As a result, it became easier for 
teachers to understand and become adept at SLIC instruction and shortened the learning 
curve for new teachers and this, in turn, facilitated classroom implementation. It is likely 
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that students attending SLIC classes in the fourth year had a richer literacy learning 
experience and more fully developed SLIC instruction than those attending classes in 
Year 1. 


Different elements of the SLIC Scope and Sequence were emphasized in Years 1, 2, and 
3 and there was consistency between Years 3 and 4. The use of text features to 
understand expository text was a primary element of instruction in Year 1. In Year 2, this 
expanded to other text forms and the manner in which meaning is communicated in these 
forms. In Year 3”, teachers present in both years gave increased attention to the way text 
forms are used by authors to communicate meaning and increased students’ use of 
literacy journals. Ratings for teachers’ use of process questioning increased among those 
who were measured over this time period. 


There was some change from Year 2 to 3 in the approaches to increasing literacy among 
English Learners and the focus on EL needs increased, according to some. In both years, 
two schools had combined EL/SLIC programs and their SLIC teachers were trained in 
EL strategies. In Year 3, with the encouragement of a few coaches, leadership stated the 
importance of student talk as a way of improving the language skills of English Learners. 
At the same time, the effort to use differentiation as a means of meeting EL needs 
increased. However, there were no instructional strategies directed uniquely at English 
Learners as part of the program, though some teachers attempted to introduce instruction 
they believed was helpful. Teachers and coaches generally indicated that SLIC is an 
effective approach to EL instruction, while a number expressed the view that it is not 
sufficient. Teachers at some sites felt freer to incorporate non-SLIC strategies than did 
others, and some believed such additions were discouraged. 


The interest in instructional needs of EL students was reflected in surveys, and SLIC 
teachers responding to the survey statement that they believe their “curriculum and/or 
instructional strategies are specifically adapted to the needs of English learners” offered a 
higher average response in Year 4 (3.54, N=13, SD=.887) than in Year 3 (3.08, N=13, 
SD .76) or Year 2 (2.14, N=7, SD=.69).'° The response to this question among all 
teachers was higher in Year 2 (2.9) than among SLIC teachers, but in Year 3 and Year 4 
SLIC teachers’ response was higher, on average, than that offered by all teachers (2.81 in 
Year 3, 2.74 in Year 4 among content teachers and 2.85 for all teachers). 


* Observations with SLIC teachers compared in both Year 2 and Year 3 include four time periods, winter 
and spring of Year 2 and fall and winter of Year 3: The observation protocol was changed before and after 
this and teacher practice cannot be compared. 

'° This should be interpreted with caution, since not all SLIC teachers answered the survey. 
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Confounds 


There are two main structural confounds associated with SLIC implementation, and they 
applied to all four years of the program. The first is that the whole school program is 
essentially the same model as the targeted program, and therefore higher implementation 
of the whole school program means not only that targeted students will have SLIC 
literacy instruction reinforced in all classes, it also raises the likelihood that control 
students will be exposed to the same program, which may make impacts harder to detect. 


A related confound is that three intervention schools in Year 3 and four schools in Year 4 
had other literacy classes (called Literacy Advancement Academies, LAA) in addition to 
SLIC, which also has the district title of a Literacy Advancement Academy. There were 
also alternate literacy classes in some schools in Year 1 and Year 2. The form and content 
of these LAA classes varied by school and by year and it is not certain that these classes 
represent a genuine confound. Based on District data, in Year 3 there were 102 students 
in LAA classes at the high school, and 43 and 27 at the two middle schools, and some 
fraction of these were control students. In Year 4 there were 32 students. Some LAA 
teachers participated in SLIC professional development as part of the whole-school SLIC 
program. For example, in Year 3, eight of the nine LAA teachers provided written 
comments on the teacher survey, and half of them noted some change in their instruction 
related to SLIC'’. One teacher noted “more emphasis on text forms and features,” 
another simply stated “I have tried the strategies in my classroom,” another commented 
that it “has put an emphasis, department/school wide, on scaffolding and student needs 
which has changed how I’m able to instruct my AVID students” and another said “I use 
deconstruction questions and strategies that I learned in the professional development 
days.” In contrast, another teacher stated that he/she has always used strategies in 
teaching, and another said he/she has had little contact with SLIC teachers. The 
assignment of control students to alternate LAA classes is quite understandable, and it 
may be that the number of control students so assigned is too small to affect the 
measurement of SLIC impacts. Nonetheless, it is possible that there is a greater gap in 
gains made by SLIC and control students at schools that have no LAA classes than at 
schools that have control students in LAA classes. In other words, SLIC students may 
show smaller relative gains at schools in which control students are attending other LAA 
classes. Furthermore, it is likely that as the whole-school program reaches more teachers 
and students at intervention schools, the differences between treatment and control 
students will be harder to detect. 


'' The question states “Has your instructional practice changed as a result of your school's participation in 
SLIC? If so, how? If not, why?” 
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Implications for Impact Analysis of Targeted Intervention 


The implementation findings have implications regarding impacts on student outcomes. 
First, implementation in some classrooms is sufficiently high to support impacts on 
students, and the consistency and depth of SLIC instruction has increased since the first 
and second year. Yet, once fidelity scores are averaged across classrooms at a school, the 
likelihood of seeing impacts diminishes and the chance of school-level impacts is less 
assured. 


Second, while the number of material scaffolds has increased and the delivery of 
professional development shifted from year to year, the ideas, curriculum, and pedagogy 
advocated by the developers have not changed but rather have become more thoroughly 
communicated and therefore easier for teachers to implement. Thus the validity of 
combining data for student impacts across years does not at present appear threatened by 
the changes in the program, and a more thorough implementation and a more easily 
transferable program is to be desired. 


Third, since students can learn the same SLIC skills and strategies in 7 grade as in 10" 
and since students at each grade level are exposed to the full SLIC program, there is no 
reason to anticipate that students remaining in the program longer will perform better 
than those testing out after a year or two, except as a result of additional practice and 
exposure to a more fully implemented program. Given the increase in instructional 
resources in Year 3, it is possible that students attending in Years 3 and 4 may have 
received stronger literacy support. And, since there were no new random assignments in 
Year 4, it might be more difficult for the program to show gains in Year 4 than in Year 3, 
even with more consistent fidelity across sites. 


It is unclear whether students are aware that they have been in a support class for multiple 
years and, if so, how they perceive that fact and what impact it might have on their sense 
of efficacy and motivation. 


The influence of confounds on measurement of impacts is unknown but should be noted. 
First, the whole school program is intended to reach all students and offers the same 
classroom practices as SLIC and, second, some control students are in supplementary 
literacy classes and their literacy teachers have received SLIC professional development 
and about half those teachers claim it has change their practice. Thus higher 
implementation of the whole school program might be associated with lower detectable 
impacts of the targeted intervention program. 
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IV. EVALUATION OF THE IMPACTS OF THE TARGETED 
INTERVENTION: YEAR 4 


Study Design 


Eight cohorts of students randomly assigned into treatment and control groups will be tracked 
longitudinally over the course of the project. Treatment students will participate in the SLIC 
targeted intervention classes for successive years (up through Grade 10) until they no longer meet 
eligibility criteria for SLIC. After the 10th grade, informal supports (e.g., literacy coaches 
monitor students’ academic progress, tutorials as needed) will likely be available. Control 
students will participate in the district’s ongoing literacy program and enroll in their normal 
elective class. These cohorts are outlined in Table 10. 


Table 10. Student Cohorts in the SLIC Program Implementation Schedule 
Year | Year 2 Year 3 Year 4 Year 5 


S06 F0O6 S07 FO7 S08 FO0&8 S09 FOO S10 F10 


Planning x x x 


Grade 7* 1 1 5 5 

Grade 8* 2 2 1,6 16 5 5 
Grade 9* 3 3 2,7 2,7 1,6 1,6 
Grade 10* 4 4 3,8 330: O25 2,7 
Grade 11 4 4 3,8 3,8 
Grade 12 4 4 
Follow-Up 


* Targeted SLIC program implementation 


The targeted SLIC program is designed as a multi-year intervention with students 
receiving up to three years of additional intensive literacy instruction. Students no longer 
defined as struggling will no longer receive the targeted intervention. Those students will 
continue to be followed as part of the treatment group, as part of the Intent-To-Treat 
(ITT) model. The targeted intervention study examines differences in treatment and 
control group outcomes from year to year cross-sectionally and longitudinally. 
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It is not the intention of the research design for control group students to receive the 
targeted intervention in later years of the study as long as data are continuing to be 
collected on the two groups of students. In other words, eligible students randomly 
assigned into the treatment or control groups will remain in their respective groups for the 
duration that they are eligible to participate in the study. Students in treatment groups will 
not move into the control group in later years, nor will control group students receive the 
treatment in later years. 


Sampling Plan 


In Years 2 and 3, the eligibility criteria for students to qualify for the targeted SLIC 
intervention were broadened to include more English learners in the sample. Thus 
students were deemed ELIGIBLE striving readers if they met at least one of the 
following three criteria: 


a. California English Language Development Test (CELDT): Overall proficiency level is 
Intermediate or below (Beginning, Early Intermediate, or Intermediate). 


b. Degrees of Reading Power (DRP): Spring 2007 independent level score falls below the 
Spring grade-level performance cut point for grades 4, 5, 6 and 7, depending on the pre- 
enrollment grade of the student (7, 8, 9, and 10, respectively). 


c. California Standards Test — English-Language Arts (CST-ELA): Spring 2007 CST- 
ELA scale score is less than or equal to 333 (lower two-thirds of basic and below). 


Students were deemed INELIGIBLE for Striving Readers if they met any of the 
following criteria: 


a. Pre-enrolled in a high school and non-diploma bound 


b. Pre-enrolled in a middle school and currently enrolled in a special education English 
class 


c. Pre-enrolled in the 7th grade and currently enrolled in an elementary special day class 


d. English learner with an OPL (CELDT score) of “beginning” and projected to be 
enrolled less than a year in the district by the end of the current school year (1.e., less than 
180 school days) 


e. Designated "D/HH" (Deaf/hard-of-hearing) in special education. 
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Sample Size and Power 


As noted earlier, students in grades 7-10 were assessed for eligibility as striving readers, 
whereby they had to meet at least one of the three eligibility criteria. Course enrollments 
were examined to determine whether the students were correctly placed according to their 
assignment and found that overall compliance between actual assignment and placement 
was about 92% across the 8 intervention schools in Year 2 and 89% in Year 3. Following 
is a breakdown of the students who qualified as striving readers and a summary of their 
randomization assignments (see Table 11). The number of students have steadily 
increased over time as students identified as eligible for the targeted SLIC program are 
added into the sample. In Year 4 (2009-10), for example, there were 718 eligible 
students continuing assignments to the treatment group and 715 eligible students 
continuing assignments to the control group. A detailed account of the randomization of 
students is presented in Figures 6, 7 and 8. 


Table 11. Number of Students Randomly Assigned to Targeted SLIC Intervention 
and Control, by First Year of Participation 


“YEAR.—~<‘“iti‘CS CTT aged SLIG—“‘CO;*;CControlés=~O2|©™~—~ Total 
Year1(2006-07) 80 78 158 
Year 2 (2007-08) 705 716 1481 
Year 3 (2008-09) 324 314 638 
Year 4 (2009-10) 718 715 1433 
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Assessed for eligibility 
(n=2,022) 


Excluded (n=1 ,852)* 
Randomized 


(n=170) 


Assigned to treatment group Assigned to control group 


Enrollment check Enrollment check 
not enrolled at school not enrolled at school 
(n=6) (n=6) 
enrolled in SLIC class enrolled in control condition 
(n=61) (n=69) 
enrolled in control condition enrolled in SLIC class 


(n=19) (n=9) 


Included in analysis as treatment Included in analysis as control 
(n=80) (n=78) 


*Random assignmentwas not effectively implemented at 3 of 5 Year 1 schools. Students from these schools were dropped from the 
targeted intervention impact analysis. 


Figure 6. Year 1 Random Assignment 
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Assessed for eligibility 


(n=4422) 


Randomized 
(n=1 666) 


Excluded (n=2,756)* 
ineligible (n=2,490) 
continuing (n=59) 
removed (n=137) 
exempted by school (n=70) 


Assigned to treatment group Assigned to control group 
(n=794) (n=872) 


Enrollment check 
not enrolled at school 
(n=89) 
enrolled in SLIC class 


Enrollment check 
not enrolled at school 
(n=96) 


enrolled in control condition 
(n=756) 

enrolled in SLIC class 
(n=20) 


(n=601) 
enrolled in control condition 
(n=104) 


Included in analysis as treatment 
(n=705) 


Included in analysis as control 
(n=776) 


* Continuing = continuing treatment assignmentfrom Year 1; removed=continuing SLIC or control studentat a site with unsuccessful Year 
1 random assignment. 


Figure 7. Year 2 Random Assignment 
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Assessed for eligibility 
(n=4336) 


Eligible (n=1,108) 


Continuing (a=1,178) (Note: Only included 
in two-year analyses.) 

Year 1 treatment (n=33) 

Year 1 control (n=34) 

Year 2 treatment (n=530) 

Year 2 control (n=581) 


Exempted (n=45) 
—. 


No assignment given 
(n=425)** 


Assigned to treatment group (n=324) Assigned to control group (n=314) 
Allocation | 


Enrollment check Follow-up Enrollment check 
enrolled in SLIC class enrolled in control condition 


(n=259) (n=309) 
enrolled in control condition enrolled in SLIC class 
(n=65) (a=5) 


Included in one-year analysis as Analysis Included in one-year analysis as 
treatment (n=324) L ) control (n=314) 


*Continuing = continuing treatment assignment from Year 1 or 2. 


**This category wouldinclude late-enrolling students and students whose Spring 2009 CST-ELAs made them newly eligible. 


Figure 8. Year 3 Random Assignment 


Minimum detectable effect (MDE) estimates were calculated based on the actual 
(not estimated) total number of enrolled Striving Readers and by subgroups (middle 
school students, high school, English learners. In Year 4, similar MDE estimates were 
calculated and there is sufficient power (80%+) to detect small to medium effects of .12 
to .44 for two years of participation in the targeted SLIC program’. The minimum 
detectable effects (MDE) include adjustments for baseline covariates (see Table 12). 


"2 One literacy intervention study (Greenfield, Schoenbach, Cziko, & Mueller, 2001) suggested effect sizes 
in the neighborhood of .25, based on pre-post DRP scores. 
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Table 12. Minimum Detectable Effects with 80% Power for Targeted SLIC 
Analyses (Two Year Participation) 


English 
Full Sample — Learners Middle School High School 
CST 0.12 0.18 0.17 0.19 
DRP 0.13 0.20 0.16 0.23 
CAHSEE 0.29 0.39 --- --- 
Motivation Y4 0.27 0.34 0.38 0.44 


Notes: Minimum detectable effects are calculated in two stages. First, we multiply the standard 
error of the impact estimate by the appropriate Z score (2.8). Second, we convert the result into a 
standardized effect size by dividing by the standard deviation of the control group. 


For three years of participation in the targeted SLIC program (in Year 4), assuming a 
fixed effects model, we have sufficient power (80%+) to detect a small to high effect 
sizes of .24 to .69, depending on the outcome measure (e.g., CST-ELA, DRP, CAHSEE, 
Motivation) and subgroup analysis (English Learners, middle school students, or high 
school students). The minimum detectable effects (MDE) include adjustments for 
baseline covariates (see Table 13). 


Table 13. Minimum Detectable Effects with 80% Power for Targeted SLIC 
Analyses (Three Year Participation) 


Full 

Sample English Learners Middle School High School 
CST 0.24 0.28 0.26 0.63 
DRP 0.26 0.38 0.26 --- 
CAHSEE 0.50 0.69 --- --- 
Motivation Y4 0.40 0.53 0.44 0.51 


Notes: Minimum detectable effects are calculated in two stages. First, we multiply the standard 
error of the impact estimate by the appropriate Z score (2.8). Second, we convert the result into a 
standardized effect size by dividing by the standard deviation of the control group. 


Description of the Counterfactual 


Students in the control group were subject to different kinds of treatment, depending on 
the schools they attended. At most intervention schools, control students took elective 
classes, following a ‘business as usual’ model. In other schools, control students received 
some form of literacy intervention, such as a Literacy Advancement Academy, in 
addition to ELA classes. More specific information about the counterfactual, on a year- 
by-year basis, is presented below. 
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In Year 1, for example, two of the five intervention schools had intact control classes, 
although only one of the two successfully randomized students into targeted SLIC and 
control classes. At one of these schools, the control class functioned much like a 
beginning English language development class because of the preponderance of English 
learners. The other school placed control students in English Language Arts (ELA) 
classes that used the Newbridge curriculum and, for the lowest-level striving readers, 
READ 180. In the READ 180 class, students were given both whole-group and small- 
group instruction along with independent reading time and computer-based 
individualized literacy skills practice. 


In Year 2, three of the eight intervention schools (three additional schools were added in 
Year 2) had intact control classes. One school continued to offer READ 180 for its lowest 
level students. In another school, students in the control condition took a supplemental 
drama class focused on English listening/speaking standards, which was effectively an 
extension of their English class. This allowed students to have twice the time with their 
English teacher. One other school provided supplemental ELA support for their control 
students. Teachers created their own curriculum for the class, often extending time on 
ELA lessons. This class used the Jamestown Reading Navigator as part of their 
curriculum. 


In Year 3, one of the eight intervention schools continued to offer intact literacy classes 
to control students. The school that used the Jamestown Navigator program in Year 2 
continued the program into Year 3. It used the Jamestown Reading program about 25 
minutes/day), along with previews/reviews of grade level language arts content and 
systematic English Language Development (ELD) instruction. The remaining two 
schools that offered literacy instruction in Year 2 no longer provided these services in 
Year 3. More specifically, the school which offered READ 180 for its lowest level 
students in Years | and 2 closed and was replaced by another school which did not offer 
the READ 180 program. Additionally, the school in Year 2 which offered the 
supplemental drama/English class did not offer this in Year 3. 


In Year 4, four of the eight intervention schools (three middle and one high school) 
offered literacy classes to students who were not enrolled in SLIC. Classes included 
Reading Apprenticeship and a support class for the California High School Exit Exam at 
the high school, and a full class period of Jamestown Navigator at one of the middle 
schools. 


In summary, the pattern differs somewhat by school and by year. Half the schools in 
Year 4 offered literacy support classes in addition to SLIC, while the majority of 
intervention schools in Years 1, 2, and 3 did not offer an intact control class (i.e., three of 
five schools in Year 1; five of eight schools in Year 2; and seven of eight schools in Year 
3). Rather, students randomized into the control group enrolled in elective classes, 
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following a “business as usual’ model. Students enrolled in the SLIC intervention classes 
were not able to enroll in electives so elective classes (e.g., Spanish or other foreign 
language, band) were effectively the control condition. As a consequence, SLIC students 
at these schools spent more time on literacy than did the control students, in contrast to 
the situation at the schools which offered literacy interventions, of different sorts, to both 
SLIC and control students. 


Data Collection Plan 


Several instruments were used in the targeted SLIC intervention evaluation: 


Degrees of Reading Power (DRP). The Degrees of Reading Power is a direct measure of 
reading comprehension. Standard DRP tests measure how well students understand the 
surface meaning of what they read. Tests consist of nonfiction paragraphs and/or 
passages on a variety of topics. Words have been deleted and students are asked to select 
from a set of multiple-choice options the correct word for each deletion in text. The DRP 
are measured on a text difficulty scale. Scores are criterion-referenced and can be 
reported in national percentiles, stanines, and Normal Curve Equivalents (NCEs). They 
run on a scale from 15- to 99+ and reflect the most difficult text a student can read with 
comprehension and map on directly to the DRP readability scale where test scores can be 
interpreted directly in terms of the difficulty of materials students are able to read. 


The DRP Handbook (2000) notes that KR-20 reliability coefficients (internal 
consistency) for grades 6-12 are high, ranging from .94-.97, with small standard errors of 
measurement (SEM range 3.0-3.4, raw scores, 70 item tests). Test-retest data are not 
available, however, alternate form reliability is high (r=.87-.91). Rasch analyses suggest 
no evidence of test bias, with models fitting equally well for students by ability level, 
racial/ethnic subgroup, gender, and by socioeconomic status. The DRP is administered to 
students in Grades 7-10 in the 16 Striving Readers schools in early fall and late spring to 
provide pre-and post- DRP assessment data. 


California Standards Test (CST). The CST measures student performance against 
California’s content standards in English Language Arts, Mathematics, Science and 
History/Social Science. The content standards are grade- and course-specific and specify 
what each student should know and be able to do. The CST is administered annually 
every spring to students as part of the Standardized Testing and Reporting (STAR) 
program, California’s statewide testing and accountability program. The CST-English 
Language Arts Test is administered to all students, in grades 2-11, including English 
learners regardless of the length of time they have been in U.S. schools or their fluency in 
English, and students with disabilities who receive special education services. 
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Results of the CSTs are reported using scale scores, which in 2007 ranged from 150 to 
600 for each test. In addition, the CST test performance for each student is categorized 
into one of the following proficiency levels: far below basic, below basic, basic, 
proficient, and advanced. The state’s target is for all students to score proficient or 
advanced. For all CSTs, the minimum scale scores defining basic and proficient are 300 
and 350, respectively. The minimum scale scores defining below basic and advanced 
vary by CST. 


Considerable evidence of reliability and validity for the California Standardized Tests is 
documented in the CST Technical Report (Educational Testing Service, 2008, February). 
All grade-level CSTs for ELA, Math, and History/Social Science were highly reliable 
with Cronbach’s alpha reliabilities ranging from 0.91 to 0.95. Reliabilities for higher- 
level CSTs for Mathematics and Science were also high, ranging from 0.89 to 0.93. The 
reliabilities were highly consistent across all CSTs within the subject areas of ELA and 
History/Social Science and across grade-level CSTs for Mathematics. 


The CST-ELA for grades 6-11 consists of 75-items and includes 5 strands/reporting 
clusters linked to the California Language Arts standards: Word Analysis, Reading 
Comprehension, Literary Response and Analysis, Writing Strategies, and Written 
Conventions. Internal consistency is high with Cronbach’s alpha ranging from .93-.95, 
with SEMs (raw scores) ranging from 3.63 to 3.90. 


Content validity evidence is based on the Spring 2007 test assembly process, and 
numerous reviews are conducted by experts in their designated areas from both the 
California Department of Education (CDE) and Educational Testing Service (ETS). For 
these reviews, ETS senior content staff worked directly with CDE content consultants. 
The CDE content consultants each have extensive experience in K-12 assessments, 
particularly in their subject of expertise, and many are former teachers. CST items were 
developed to align with the content standards that are representative of the broader 
content domains: English Language Arts, Mathematics, History/Social Science, and 
Science. Thus, the content-related evidence of validity concerns the extent to which the 
test items represent these specified content domains and cognitive dimensions. Content 
validity also provides information about how well an item measures its intended 
construct. 


Convergent validity is based on a study relating the CAT/6 Survey tests and the CSTs 
using the 2004 data when a full complement of CAT/6 Survey tests were administered to 
students in grades 2-11 throughout California. Other validity studies have been conducted 
by the CDE and local districts throughout California. 


In addition to content, all items are reviewed and approved to ensure their adherence to 
the principles of fairness and to ensure no bias exists with respect to characteristics such 
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as gender, ethnicity, and language. Differential item functioning (DIF) analyses suggest 
no evidence of test bias. Additional information about the California Standards Test 
(CST) is available from the California Standards Tests (CSTs) Technical Report Spring 
2007 Administration (Educational Testing Service, 2008). 


California English Language Development Test (CELDT). The CELDT assesses the 
construct of English language proficiency, as defined by the California ELD standards. 
The overall proficiency construct is composed of the four domains of Listening, 
Speaking, Reading, and Writing. The CELDT is criterion-referenced assessment and is 
administered to all students in grades 2-12 who have been identified as English learners 
and to students who are new to the district whose family uses a home language other than 
English. 


Cronbach’s alpha reliability coefficients for the CELDT 2006-07 Edition (Form F) range 
from 0.81 to 0.91 across grades 6-12 and across the four domains (Listening, Speaking, 
Reading, Writing). The range of standard errors is between 1.3 and 2.52 points across 
grades 6-12 and subject areas in raw score units. The lowest obtainable scale score 
(LOSS) for all 4 domains combined (CELDT Overall) was 248, and the highest overall 
scale score (HOSS) was 761, varying by grade and individual domains. Scale scores are 
divided into five performance levels -- Beginning, Early Intermediate, Intermediate, Early 
Advanced, and Advanced -- which are the same levels used in the California ELD 
Standards. Each student’s performance on the CELDT is then defined by performance 
levels on the test scale delineated by cut scores. 


Validity evidence includes an evaluation of the alignment of the Form E California 
English Language Development Test (CELDT, CTB/McGraw-Hill, 2005) to the 
California English Language Development (ELD) standards. All ELD standards at four 
grade spans (K—2, 3 —5, 6 — 8, and 9 — 12) and the 396 CELDT 2005-06 Edition (Form 
E) items were used to evaluate alignment. Most ELD and ELA standards were ratable 
(greater than 90%). There were variable degrees of alignment between ELD standards 
and the CELDT depending on language demands and proficiency levels (i.e., as defined 
by the ELD standards). Strongest alignment appeared in items in the Reading and Writing 
sections on the language-demand dimensions. Reading and Speaking items generally 
showed moderate to strong alignment across sections of the test on all dimensions (_e., 
ratability, modality, complexity, and language demands), especially in the lower two 
grade spans. Items in the Writing sections were weakly aligned on the modality and 
complexity dimensions, and Listening items showed the weakest alignment on all 
dimensions. The 6 — 8 grade span showed the weakest alignment of the four spans, 
especially on complexity and language-demands dimensions. Additional information 
about the CELDT is available from the Technical Report for the California English 
Language Development Test (CELDT), 2006-07 Edition (Form F). (CTB/McGraw-Hill, 
2007). 
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California High School Exit Examination (CAHSEE). The purpose of the CAHSEE is to 
significantly improve pupil achievement in public high schools and to ensure that pupils 
who graduate from public high schools can demonstrate grade level competency in 
reading, writing, and Mathematics. All students must pass the CAHSEE, as well as meet 
district’s requirements for graduation, in order to receive a high school diploma. 


The CAHSEE offers an examination in English Language Arts (ELA) and an 
examination in Mathematics. The ELA examination measures reading and writing skills 
as defined by the State Board of Education (SBE) standards through grade 10. The 
reading portion of the examination covers topics such as vocabulary and informational 
and literary reading. The writing portion of the examination covers topics such as writing 
strategies, applications, and conventions. The ELA examination consists of 72 multiple- 
choice (MC) questions and one constructed response (CR) item. The CR item is related to 
a literature or informational passage or is a written response to a writing prompt. The 
CAHSEE Mathematics examination measures standards adopted by the SBE through 
Algebra I. It covers topics such as statistics, data analysis and probability, number sense, 
measurement and geometry, algebra, and mathematical reasoning. There are 80 
operational multiple-choice (MC) items in each Mathematics form and 72 MC and one 
constructed-response (CR) item in each ELA form. The scale range is 275-450. (For 
more information about the CAHSEE, see http://www.cde.ca.gov/ta/tg/hs/overview.asp). 


The CAHSEE is administered to students in grades 10-12, and is typically administered 
seven times per year between July and the following May to allow several testing 
opportunities for those students who have not yet passed the examination). Tenth grade 
students take the CAHSEE for the first time in March. If they do not pass one of the 
CAHSEE sections (English Language Arts or Mathematics), then they re-take that 
portion. 


Reliability indices are based on the multiple 2006-07 administrations of the CAHSEE to 
California students, predominately first time 10th grade examinees along with those who 
did not pass one or both CAHSEE content areas at the first attempt. These statistics 
indicate satisfactory performance of the test form constructed for these examinations. 
Reliability indices range from .85-.95, and the raw score Standard Error of Measurement 
ranged from 3.68 to 4.43 across the 2006-07 CAHSEE administrations. Reliabilities 
(REL) and standard error of measurement estimates (SEMs) are reported for subgroups 
(i.e., gender, ethnic, accommodation, non-accommodation, English learner and English 
proficient groups) from the February and March 2007 census administrations, where 
larger case counts were available. While there is some variation within some subgroups 
(for example, SEMs are higher for the English Learner group than the English Proficient 
group), all reliabilities for both the ELA and Mathematics sections are greater than or 
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equal to 0.88. All items included on operational test forms had been evaluated for bias 
and sensitivity and for alignment of the content standards. 


During the seven administrations, the decision accuracy for ELA at the Pass/Not Pass 
level ranged from 0.91 to 0.94, while the decision accuracy for Mathematics at the 
Pass/Not Pass level ranged from 0.91 to 0.94. The decision consistency for ELA at the 
Pass/Not Pass level ranged from 0.87 to 0.91, while the decision consistency for 
Mathematics ranged from 0.88 to 0.92. The magnitudes of the numbers (0.87 or above) 
reflect a high level of accuracy and consistency in the student classifications. Additional 
information about the CAHSEE is available from the CAHSEE Technical Report, July 
2006-May 2007 Administrations (Educational Testing Service, 2008). 


San Diego Striving Readers/SLIC Assessments (SDUSD/Education Associates/BEAR). 
The diagnostic (pre), benchmark, and post-assessments are intended to be closely aligned 
to the SLIC curriculum that they reflect literacy strategies and skills that students should 
learn as part of the California English Language Arts standards. The diagnostic 
assessments are administered at the beginning of the school year to all students in the 
intervention schools for baseline assessment (pre-test in September), and a comparable 
outcome assessment is administered to all students at the intervention schools at the end 
of the school year (post-test in June). SLIC students take additional benchmark 
assessments periodically during the school year, about every 2-3 months, focusing on 
specific text types (expository, persuasive, and narrative). 


Results from the SLIC assessments provide both qualitative information on meaningful 
levels of student proficiency, for use by teachers in guiding classroom instruction, and 
scaled interval measures of student proficiency, for comparative use in the evaluation. 


San Diego Striving Readers Teacher Survey. This survey measures how teachers are 
implementing various literacy strategies (e.g., how often they ask students to find a main 
idea in their reading, how often they ask students to preview text) in their classrooms. At 
SLIC intervention schools, teachers respond to additional closed and open-ended 
questions related to the SLIC program (e.g., strengths/weaknesses of SLIC, challenges to 
implementation). The survey is administered annually to all teachers in the study schools; 
however, the response rate varies by school. (See Appendix G.) 


San Diego Striving Readers Student Survey. This survey measures how often students 
engage in various literacy strategies (e.g., reading/writing in class, previewing text 
features, taking notes in class) in core content classes. The survey also includes questions, 
drawn from validated surveys, related to their literacy activities outside of school and 
affective characteristics, such as the students’ motivation, self-efficacy as readers and 
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writers, and school engagement. The survey is administered annually to all students in the 
study schools. (See Appendix G.) 


SLIC Classroom Observation Protocol. This instrument features a checklist of SLIC 
literacy strategies and activities that might be covered in a class lesson. It includes core 
elements of the SLIC curriculum, along with ratings of teacher quality and student 
engagement and timed observations of instructional practices. The observation instrument 
is completed during regularly scheduled classroom observations (3 times annually in at 
least one randomly selected class for each intervention teacher). (See Appendix G.) 


Schedule of Data Collection 


See Table 14 for overview of administration periods for key measures used in the San 
Diego Striving Readers Project. 
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Table 14. Overview of Study Instruments 


Instrument Target population Administration window 
Assessments 
Degrees of Reading — Students in all study Fall (October) and spring (March) 
Power (DRP) schools, grades 7-10 
California Standards Students in all study Late spring (April/May) 
Test (CST) schools, grades 6-12. 

Actual test varies by 

grade level. 
California English English learners only, Early fall (October/November), as 
Language grades 6-12 needed 
Development Test 
(CELDT) 
California High Students in grades 10-12 | Administered throughout year (Oct, 
School Exit Nov, Feb, March-first time test 
Examination takers, May, July) 
(CAHSEE) 
San Diego Striving Students in intervention Pre-test in September, post-test in 
Readers/BEAR schools, 7-10 June for all students in intervention 
assessments schools. Additional benchmark 


Additional instruments 


San Diego Striving 
Readers Teacher 
Survey 


San Diego Striving 
Readers Student 
Survey 


SLIC Classroom 
Observation Protocol 


Teachers in all study 
schools, grades 6-12 


Students in all study 
schools, grades 6-12 


Randomly selected SLIC 
classrooms (one per 
teacher/grade level) 


assessments varying by text form 
administered periodically (about 
every 2-3 months) to SLIC students 
only. 


End of school-year (May/June) 


End of school-year (May/June) 


Completed by evaluation team 
during regularly scheduled 
classroom observations (3x/year) 
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Summary of Analytic Approach to the Impact Analysis 


The targeted SLIC evaluation analyzed student outcomes using two-level hierarchical 
models (students nested within 8 intervention schools) for both one year and two year 
participation in the SLIC intervention program. Students in SLIC classes were compared 
to students in control groups on selected outcome indicators (e.g., CST-ELA, DRP, 
CAHSEE-ELA, self-reported level of motivation), controlling for key confounding 
variables (e.g., parent education, student gender, English learner status, grade level, pre- 
test scores on outcome variable, and study year). The targeted SLIC evaluation included 
7", 8", 9" and 10" graders. All covariates had less than 10% missing data except for 
parent education (50% had “unknown” code). Due to small sample size the “unknown” 
education category was included in the analyses. Additional analyses were conducted by 
omitting the “unknown” parent education code and none of the results changed. 
Subgroup analyses were also conducted for English learners, middle school students, and 
high school students on the same selected outcome indicators controlling for the 
confounding variables outlined above. 


Additional analyses were conducted to examine SLIC student performance based on 
specific SDUSD Performance Goals, separate from the Intent-to-Treat model. These are 
presented in Appendix C. 


Description of the Year 1-4 Samples 


Schools and Targeted Intervention Study Samples 


In Year 2, targeted SLIC classes expanded from five schools to eight schools. In Year 3, 
seven of eight schools continued in the study, and an eighth replaced a Year 2 school in 
the same location, bringing in a new focus (on creative and performing arts), many new 
students, and mostly new teaching staff. While 2 of 5 schools successfully completed 
random assignment in Year 1, all 8 schools did so in both Year 2 and Year 3. Project- 
wide compliance with random assignment in Year 2 was 92%, with every school having 
a compliance rate above 80%. Compliance in Year 3 was 89%, with one school at 71%. 


Classrooms and Personnel 


In Year 1, the two schools that successfully carried out random assignment had three 
targeted intervention classes. One school had two intervention classes with 22 students 
each, taught by the same teacher, and control students were enrolled in an alternative 
literacy intervention. The other school had one intervention class with 26 students and no 
control class for control students. In Year 2, there were a total of 814 students enrolled in 
42 intervention classes in 7th through 10th grade at the eight treatment schools. At the 
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end of Year 1, there were a total of seven intervention teachers, three of which joined the 
20 Year 2 intervention teachers. Continuity of intervention teachers was much better in 
Year 3, with only two new teachers joining the intervention teaching staff of 16. These 
teachers taught a total of 41 intervention classes with a total of 782 students. 


Students not enrolled in the intervention class are generally enrolled in an elective class 
instead, with a minority of control students in alternative literacy interventions. In Year 1, 
two of five intervention schools had alternative literacy interventions. In Years 2 and 3, 
three of eight schools had alternative literacy interventions for students in the control 
group. In Year 3, one of the alternative intervention classes was taught by a previous 
study intervention class teacher which may have influenced instruction. 


In Years | and 2, one SLIC coach (officially, “SLIC resource teacher’’) was assigned to 
one school each to work with the intervention teachers on matters ranging from SLIC 
pedagogy and curriculum to assessment of individual students’ needs. Of the five Year | 
coaches, three joined the eight Year 2 coaches, although one was on leave for a portion of 
the year. One of the program leaders and a developer took on some of this work during 
the coach’s absence. All of the coaches had prior teaching experience. In Year 3, this 
structure changed significantly. In general, coaches took on less responsibility for 
working with targeted intervention teachers than in Year 2. This reflected both the lesser 
need of some targeted intervention teachers as they gained experience and the shifting of 
responsibility for the targeted and whole-school intervention, with program leadership 
taking on much more of former and coaches taking on much more of the latter. At two 
sites, intervention class teachers got almost none of their support from coaches. At three 
other sites, the amount of support they got from coaches diminished substantially relative 
to Year 2. Some, but not all, of this decline was offset by program leadership’s increased 
work with intervention teachers. 


Students 


In Year 1, random assignment was successful at two sites, with 73 students assigned to 
treatment and 72 to control. At 3 other sites, random assignment was not successful. 
Students enrolled in SLIC in Year 1 at these sites were deemed ineligible for inclusion in 
the study sample in Year | and beyond, but were kept in the intervention class, if eligible. 
In Year 2, there were 758 assignments to the treatment group and 777 to the control 
group, while there were 887 assignments to treatment and 929 to control in Year 3. 
Among Year 3 treatment assignments, 530 were originally given in Year 2, and 33 were 
given in Year 1. In the control group, 581 were from Year 2, and 34 were from Year 1. In 


81 


Year 4 there were no new random assignments’». Students with assignments from 
previous years are kept in the analysis sample regardless of their eligibility.“ 


Following a longitudinal intent-to-treat model, student outcomes are analyzed according 
to their original assignments, regardless of their actual enrollment in the treatment class. 
In Year 2, 110 students with treatment assignments were not enrolled in the intervention 
class, primarily due to scheduling conflicts, while 20 students with control assignments 
were enrolled in the intervention class. In Year 3, 259 of 324 students first assigned to the 
treatment group ended up in the intervention class, while 309 of 314 control students 
were in the control condition. 


In both Year | and Year 2, Hispanic/Latino students constituted the largest racial-ethnic 
group in the targeted intervention at both sites with successful random assignment, 
representing 60% of students overall, while African-American students represented 18% 
of all targeted intervention students. In Year 2, there was a somewhat higher percentage 
of Hispanic/Latino students in the treatment group than in the control group (66% and 
60%), while relatively fewer African-American and White students were in the treatment 
group. These differences diminished in Year 3: 68% of the treatment group and 65 
percent of the control group, were identified as Latino. While cross-group variation in 
ethnicity was marginally significant in Year 2, it was non-significant in Years | and 3. 


Sizable shares of students in all years and groups were English learners. In Year 1, there 
was a significantly higher percentage of English learners in the treatment group than in 
the control group (46% and 24%). Treatment-control differences in the percentage of 
English learners were non-significant in both Year 2 and Year 3, with 35% in Year 2 and 
45% in Years 3 and 4. 


Targeted intervention students went from being roughly balanced by gender in Year | to 
including slightly more male students in Year 2, with 55% male students overall in Year 
2. The overall sample was 54% male in Year 3. In all years, treatment-control differences 
in gender were non-significant. 


By design, students in both the treatment and control groups scored low on tests related to 
reading ability, particularly on the Degrees of Reading Power (DRP) test. In Year 1, the 
percentage of students in the treatment and control groups at both sites scoring at the 
equivalent of two grade levels below their current grade level or lower on the DRP pre- 
test ranged from a low of 80% to a high of 90%. In Year 2, about one third of students in 


'’ There were 623 continuing students in grades 8-11, with 135 in grade 8, 131 in grade 9, 177 in grade 10, 
and 180 in grade 11. There were 168 students in the 3" year of the intervention and 566 in the 2™ year of 
the intervention. 

'’ Year 1 control students received new random assignments for Year 2, but are treated as continuing 
control assignments if they were reassigned to control. 
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both the treatment and control groups scored “far below” grade level or lower on the 
DRP. This increased to slightly over half (55%) of the sample in Year 3. There was no 
significant difference between treatment and control group pre-test scores in Years 2, 3 
and 4. 


A majority of students also tested below “Basic” on the California Standards Test- 
English Language Arts in Year 1. In Year 2, with a larger frame of schools and grades, 
the proportion of students below "Basic" was just under half in both treatment and control 
groups. In Year 3, 49% of the treatment group and 50% of the control group was below 
basic. In both Years 2 and Year 3, there were no significant differences between 
treatment and control groups in pre-test scores. 


In Year 1, special education students comprised 22% of the targeted intervention sample. 
In Year 2, they were 19% of the sample, including 15% of the middle school and 23% of 
the high school sample. In Year 3, they were 19% of both the treatment group and the 
control group, with 17% overall in middle school and 21% in high school. While 
information on disability types was not available in Year 1, students with speech or 
learning disabilities were 15% of the Year 2 sample and 16% of the Years 3 and 4 
samples, with 11% of the middle school and 18% of the high school sample in Year 2 
and, in Years 3 and 4, 14% in middle and 18% in high schools. 


Impacts on Targeted SLIC Students 


This section displays the student outcome results for two and three years participation in 
the targeted SLIC program for all students identified as eligible striving readers 
(randomly assigned students versus their control counterparts). The analyses were first 
conducted for the full sample of eligible striving readers, based on the two and three 
years of participation in the targeted SLIC intervention. Additional analyses were 
conducted to examine student performance by subgroups of English learners only, middle 
school students, and high school students. 


Two Year Participation” 


Student reading skills were measured using assessments already in place by the San 
Diego Unified School District (SDUSD). Analyses were run to determine what 
difference, if any, the targeted SLIC program had on students’ performance on two 
reading assessments: Degrees of Reading Power (DRP) and California Standards Test — 


'S For the two year participation analyses the outcome measures were post test scores one year later than 
their assignment year. For example, for Y1 assignments we used Y2 post test scores, and for Y2 
assignments we used Y3 post test scores as outcome measures. 
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English Language Arts (CST-ELA). Analyses were not run with Advanced Placement 
course pass rates as an outcome measure due to a small sample size. 


Other student academic outcomes were measured to assess the broader impact the 
targeted SLIC program may have had on student performance. First, performance on the 
California High School Exit Examination (CAHSEE) was measured. Finally, student 
motivation was measured using a composite scale constructed from student responses to 
twelve survey questions (see Appendix B). All items ranged from “0” (not true) to “4” 
(very true), so that the resulting scale ranged from 0 (not at all motivated) to 48 (very 
motivated). Principal Component analysis was used to construct this latent variable. For 
list of the one component loadings see Table B1 (Appendix B). The scale was highly 
reliable (Cronbach’s Alpha=.873), suggesting that the questions measured a single, 
unidimensional latent variable. Two motivation outcome scales were created — one for 
Y3 and another for Y4. Separate regression analyses were conducted on Y4 motivation 
outcome using the Y3 motivation scale as a control. 


Examination of the mean pre-test scores for both the targeted SLIC students (treatment) 
and control students suggest that both groups were comparable at baseline, and that the 
control students were scoring slightly higher on their post-tests (following two years of 
participation in targeted SLIC) than their control counterparts (see Table 15). 


Table 15. Mean Pre/Post Assessment Scores for Targeted SLIC and Control 
Students (Two Year Participation) 


Control Treatment Total 
Pre Post Pre Post Pre Post 


CST-ELA 299.42 307.67 298.96 308.17 299.19 307,92 
(27.09) (38.99) (28.29) (40.09) (27.69) (39.52) 


n=792 n=792 n=782 n=782 n=1574 n=1574 
DRP 38.81 47.42 38.74 48.29 38.77 47.86 

(10.43) (10.97) (10.31) (10.61) (10.37) (10.79) 

n=591 n=591 n=587 n=587 n=1178 n=1178 


Note: Each cell includes the mean, standard deviation and sample size. CST-ELA=California 
Standards Test-English Language Arts; DRP=Degrees of Reading Power. 


Table 16 displays raw and regression-adjusted means for treatment and control groups, 
the differences in the regression adjusted means (treatment minus control), the 
standardized effect size (SES) of the treatment, and probabilities for the null hypothesis 
that the treatment and control groups were equal. The SES was calculated for each 
regression model by dividing the difference in regression-adjusted means by the standard 
deviation of the control group. 
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No significant differences between treatment and control groups were found for CST- 
ELA CAHSEE, DRP or student motivation after controlling for student’s pre score, 
gender, socioeconomic status'®, and grade level. More detailed results for each regression 
analysis and National Curve Equivalents (NCEs) for adjusted CSTs are given in 
Appendix D. 


Table 16. Estimated Effect of Targeted SLIC Intervention on Student Outcomes for 
All Eligible Striving Readers (Two Year Participation) 


Raw Means Regression-Adjusted _ Diff Control SES p- 
Means Group value 
SD 
Control Treatment Control Treatment 

CST 307.67 308.17 310.62 311.02 0.40 38.99 0.01 0.82 
DRP 47.42 48.29 47.51 48.49 0.89 10.97 0.08 0.13 
CAHSEE 354.81 354.50 360.51 359.58 -0.93 26.95 0.03 0.76 
Motiv. (Y4) 33.98 34.13 33.85 34.02 0.17 8.99 0.07 0.85 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted 


means by the standard deviation of the control group. CST-ELA=California Standards Test-English 
Language Arts; DRP=Degrees of Reading Power, CAHSEE=California High School Exit Exam. 


Three Year Participation” 


Examination of the mean pre-test scores for both the targeted SLIC students (treatment) 
who had participated for three years and control students, suggest that both groups were 
comparable at baseline and on their CST-ELA and DRP post-tests (see Table 17). 


'® Since over 50% of the targeted sample had an “unknown” code for parent education, for socioeconomic 
status we used data on percent with BA degree by student home zip code (source: http://www.city- 
data.com/) in the hierarchical linear modeling (HLM) analyses. 

'’ For the two year participation analyses the outcome measures were post test scores one year later than 
their assignment year. For example, for Y1 assignments we used Y2 post test scores, and for Y2 
assignments we used Y3 post test scores as outcome measures. 
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Table 17. Mean Assessment Scores for Targeted SLIC and Control Students (Three 
Year Participation) 


Control Treatment Total 

Pre Post Pre Post Pre Post 
CST-ELA 299.14 312.66 299.37 311.49 299.25 312.07 
(26.18) (45.89) (28.58) (45.79) (27.39) (45.81) 

n=301 n=301 n=305 n=305 n=606 n=606 

DRP 38.76 50.87 38.04 50.98 38.38 50.93 
(9.46) (11.03) (9.64) (9.01) (9.54) (10.02) 
n=156 N=156 n=168 n=168 n=324 n=324 


Note: Each cell includes the mean, standard deviation and sample size. CST-ELA=California 
Standards Test-English Language Arts; DRP=Degrees of Reading Power. 


Table 18 displays raw and regression-adjusted means for treatment and control groups, 
the differences in the regression adjusted means (treatment minus control), the 
standardized effect size (SES) of the treatment, and probabilities for the null hypothesis 
that the treatment and control groups were equal. The SES was calculated for each 
regression model by dividing the difference in regression-adjusted means by the standard 
deviation of the control group. 


No significant differences between the targeted SLIC students (treatment) and control 
students were found for CST-ELA scores, CAHSEE, DRP, or student motivation. More 
detailed results for each regression analysis and NCEs for adjusted CSTs are given in 
Appendix D. 


Table 18. Estimated Effect of Targeted SLIC Intervention on Student Outcomes for 
All Eligible Striving Readers (Three Year Participation) 


Raw Means Regression-Adjusted _ Diff Control SES p- 
Means Group value 
SD 
Control Treatment Control Treatment 

CST 312.66 311.49 318.47 316.94 -1.53 45.89 -0.03 0.70 
DRP 50.87 50.98 51.38 51.66 0.28 11.03 0.25 0.80 
CAHSEE 353.56 352.67 953.12 353,26 0.14 26.76 0.01 0.98 
Motiv. (Y4) = 34.18 35.33 34.12 34.99 0.87 8.63 0.10 0.51 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted 


means by the standard deviation of the control group. CST-ELA=California Standards Test-English 


Language Arts; DRP=Degrees of Reading Power, CAHSEE=California High School Exit Exam. 
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Subgroup Analyses 


English Language Learners 


Additional HLM analyses were conducted to determine whether participation in targeted 
SLIC classes had an impact on outcomes for English learner students. These models 
included the same dependent variables and covariates as the analyses conducted on the 
whole sample, but the sample was only the subset of students classified as English 
learners. 


The results of these analyses, displayed in Table 19, reveal few significant differences in 
the way that targeted SLIC classes affected English learner students in comparison to the 
rest of the sample. Assignment to SLIC had no significant effect on any measured student 
outcomes for the English Learners subset. English Learner subset analyses should be 
interpreted with caution, however, since statistical power is significantly reduced given 
the smaller sample of cases. 


Table 19. Estimated Effect of Targeted SLIC Intervention on Student Outcomes for 
English Learners (Two Year Participation) 


Raw Means Regression-Adjusted _ Diff Control SES p- 
Means Group value 
SD 
Control Treatment Control Treatment 

CST 299.18 300.21 300.30 301.85 1.55 38.37 0.04 0.55 
DRP 43.47 45.12 43.70 44.92 122 10.33 0.12 0.14 
CAHSEE 344.59 344.14 348.54 348.34 -0.20 28.58 0.01 0.96 
Motiv. (Y4) — 33.75 5312 33.20 34.26 1.06 9.43 0.11 0.39 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted 


means by the standard deviation of the control group. CST-ELA=California Standards Test-English 


Language Arts; DRP=Degrees of Reading Power, CAHSEE=California High School Exit Exam. 


Additionally, for three-years of participation, there are no significant differences in the way 


targeted SLIC classes affected English learner students in comparison to the rest of the sample. 


(See Table 20.) English Learner subset analyses should be interpreted with caution, since 
statistical power is significantly reduced given the smaller sample of cases. 
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Table 20. Estimated Effect of Targeted SLIC Intervention on Student Outcomes for 
English Learners (Three Year Participation) 


Raw Means Regression-Adjusted _ Diff Control SES p- 
Means Group value 
SD 
Control Treatment Control Treatment 

CST 305.04 303.57 309.92 307.92 -2.30 46.81 -0.05 0.64 
DRP 47.18 48.15 48.41 48.37 -0.04 11.11 -0.00 0.98 
CAHSEE 341.14 346.23 343.04 345.86 2.82 23.18 0.12 0.64 
Motiv. (Y4) 32.84 34.82 32.56 35.07 2.51 9:53 0.26 0.21 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted 


means by the standard deviation of the control group. CST-ELA=California Standards Test-English 


Language Arts; DRP=Degrees of Reading Power, CAHSEE=California High School Exit Exam. 


Middle and High School Students 


Analyses were conducted to detect differences in the causal effect of the targeted SLIC 
intervention between middle school and high school samples for two-years and three- 
years of participation. Differences between treatment and control groups remain 
statistically indistinguishable from zero for both middle school and high school samples 
with regards to achievement on the CST, DRP, CAHSEE and measures of student 
motivation. 


Tables 21 and 22 show the estimated effect of the targeted SLIC intervention on student 
outcomes for middle school and high school students for two years of participation, 
respectively. It should also be noted that inclusion of the student motivation scale as a 
covariate in the other above regression equations did not significantly affect findings. 


Table 221. Estimated Effect of Targeted SLIC Intervention on Student Outcomes 
for Middle School Students (Two Year Participation) 


Raw Means Regression-Adjusted _ Diff Control SES p- 
Means Group value 
SD 
Control Treatment Control Treatment 
CST 312.98 314.45 SLIT 314-73 1.96 36.80 0.05 0.43 
DRP 45.81 46.49 45.70 46.48 0.78 10.36 0.08 0.28 
Motiv. (Y4) 33.40 33.79 33.15 33.62 0.47 9.45 0.05 0.74 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted 


means by the standard deviation of the control group. N=4 middle schools. CST-ELA=California 
Standards Test-English Language Arts; DRP=Degrees of Reading Power. 
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Table 22. Estimated Effect of Targeted SLIC Intervention on Student Outcomes for 
High School Students (Two Year Participation) 


Raw Means Regression-Adjusted Diff Control SES p- 
Means Group value 
SD 
Control Treatment Control Treatment 
CST 301.49 301.11 308.47 307.08 -1.39 40.56 -0.03 0.64 
DRP 51.35 52.54 51.28 52.61 133 11.44 0.12 0.25 


Motiv. (Y4) = 35.03 34.72 30.25 34.51 -0.74 8.06 -0.09 0.61 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression adjusted 
means by the standard deviation of the control group. N=4 high schools. CST-ELA=California Standards 
Test-English Language Arts; DRP=Degrees of Reading Power. 


Finally, analyses were conducted to detect differences in the causal effect of the targeted 
SLIC intervention among middle school and high school students, for three years of 
participation in the program. Differences between treatment and control groups remain 
statistically indistinguishable from zero for both middle school and high school samples 
with regards to achievement on the CST, DRP, CAHSEE and measures of student 
motivation. Tables 23 and 24, show the estimated effect of the targeted SLIC 
intervention on student outcomes for middle school and high school students, 
respectively. It should also be noted that inclusion of the student motivation scale as a 
covariate in the other above regression equations did not significantly affect findings. 


Table 23. Estimated Effect of Targeted SLIC Intervention on Student Outcomes for 
Middle School Students (Three Year Participation) 


Raw Means Regression-Adjusted _ Diff Control SES p- 
Means Group value 
SD 
Control Treatment Control Treatment 
CST 325.56 323.70 328.43 329.24 0.81 41.97 0.02 0.85 
DRP 51.00 51.00 51.62 51.58 -0.04 10.95 -0.00 0.98 


Motiv. (Y4) 33.02 35.83 32.56 35.44 2.88 9.69 0.30 0.15 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted 
means by the standard deviation of the control group. N=4 middle schools. CST-ELA=California 
Standards Test-English Language Arts; DRP=Degrees of Reading Power. 
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Table 24. Estimated Effect of Targeted SLIC Intervention on Student Outcomes for 
High School Students (Three Year Participation) 


Raw Means Regression-Adjusted _ Diff Control SES p- 
Means Group value 
SD 
Control Treatment Control Treatment 
CST 295.23 295.90 307.68 299.94 -7.74 45.33 -0.17 0.51 
DRP NA 


Motiv. (Y4) 35.41 34.81 35.58 34.60 -0.98 912 -0.11 0.61 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted 
means by the standard deviation of the control group. N=4 high schools. NA: Not Available (no data). 
CST-ELA=California Standards Test-English Language Arts; DRP=Degrees of Reading Power. 


Treatment on Treated (TOT) Analysis using Bloom’s Adjustment 


To account for the effect of “no shows” (i.e., students who were assigned to the treatment 
but did not participate), the Bloom’s adjustment (see Bloom, 1984) was used for two 
outcome measures: California Standardized Test (CST) and Degrees of Reading Power 
(DRP), for one year, two years, and three years of participation. 


For one year participation, about 25% were non-participants (1.e., not enrolled in SLIC 
classes) and the rest (75%) attended SLIC classes at most 110 days out of 180 total 
school days (i.e., attended 110/180=61% of time or less). On the other hand, the vast 
majority (over 90%) of “no shows” for two and three years of participation were non- 
participants (1.e., not enrolled in SLIC classes), and the rest (about 10%) attended SLIC 
classes at most 61% of the time. 


Using the Bloom’s adjustment, Table 25 displays the 95% confidence interval estimates 
of the intervention (i.e., SLIC) effect and the t-statistics for the Full Sample. Results from 
the Bloom’s adjustments indicate no SLIC effect for CST for any of the study cohorts 
(one year, two years and three years of participation). However, for DRP, the results 
indicate a small SLIC (intervention) effect for one year (p<.05) and two years of 
participation (p<.05) but not for three years of participation. For one year and two years 
of participation the percentage effect of the intervention is 3.12% and 4.46%, 
respectively. 
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Table 25. Bloom’s Adjustment for CST and DRP by Year of Participation (Full 
Sample) 


(95% Confidence t-test % Program Effect 
Interval) 
CST 
One Year Participation (-2.42, 5.28) 0.73 0.47% 
Two Year Participation (-4.18, 8.40) 0.66 0.70% 
Three Year Participation (-23.14, 19.14) -0.19 -0.64% 
DRP 
One Year Participation (0.23, 2.49) 2.36 3.12%* 
Two Year Participation (0.27, 3.75) 2.26 4.46%* 
Three Year Participation (-4.57, 2.73) -0.49 -1.88% 


* p<.05. CST-ELA=California Standards Test-English Language Arts; DRP=Degrees of 
Reading Power. 


Using the Bloom’s adjustment, Table 26 displays the 95% confidence interval estimates 
of the intervention (i.e., SLIC) effect and the t-statistics for English Learners. For this 
subgroup, results from the Bloom’s adjustments indicate no SLIC effect for CST and 
DRP for any of the study cohorts (one year, two years and three years of participation). 


Table 26. Bloom’s Adjustment for CST and DRP by Year of Participation (English 
Learners) 


(95% Confidence t-test % Program Effect 
Interval) 
CST 
One Year Participation (-5.47, 4.78) -0.13 -0.12% 
Two Year Participation (-6.05, 10.90) 0.56 0.82% 
Three Year Participation (-25.64, 22.18) -0.14 -0.57% 
DRP 
One Year Participation (-0.43, 2.57) 1.40 2.68% 
Two Year Participation (-0.72, 4.09) 1.89 4.78% 
Three Year Participation (-3.60, 5.41) 0.39 2.01% 


CST-ELA=California Standards Test-English Language Arts; DRP=Degrees of Reading Power. 


Using the Bloom’s adjustment, Table 27 displays the 95% confidence interval estimates 
of the intervention (i.e., SLIC) effect and the t-statistics for Middle School students. For 
this subgroup, results from the Bloom’s adjustments indicate no SLIC effect for CST and 
DRP for any of the study cohorts (one year, two years and three years of participation). 
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Table 27. Bloom’s Adjustment for CST and DRP by Year of Participation (Middle 
School) 


(95% Confidence t-test % Program Effect 
Interval) 
CST 
One Year Participation (-3.40, 5.52) 0.47 0.35% 
Two Year Participation (-2.93, 10.65) 1.08 1.17% 
Three Year Participation (-17.17, 10.92) -0.44 -1.00% 
DRP 
One Year Participation (-1.10, 1.37) 0.21 0.32% 
Two Year Participation (-1.18, 2.52) 0.71 1.47% 
Three Year Participation (-3.95, 3.25) -0.19 -0.72% 


CST-ELA=California Standards Test-English Language Arts; DRP=Degrees of Reading Power. 


Using the Bloom’s adjustment, Table 28 displays the 95% confidence interval estimates 
of the intervention (i.e., SLIC) effect and the t-statistics for High School students. For this 
subgroup, results from the Bloom’s adjustments indicate no SLIC effect for CST and 
DRP for any of the study cohorts (one year, two years and three years of participation). 


Table 28. Bloom’s Adjustment for CST and DRP by Year of Participation (High 
School) 


(95% Confidence t-test % Program Effect 
Interval) 
CST 
One Year Participation (-5.38, 6.37) 0.17 0.16% 
Two Year Participation (-13.33, 12.50) -0.06 -0.14% 
Three Year Participation (-124.56, 167.35) 0.29 7.57% 
DRP 
One Year Participation (-1.06, 2.49) 0.79 1.48% 
Two Year Participation (-1.10, 6.19) 1.37 5.36% 
Three Year Participation Not Available 


CST-ELA=California Standards Test-English Language Arts; DRP=Degrees of Reading Power. 


Discussion 

Analyses of all eligible striving readers in the targeted SLIC evaluation found no 
significant differences in California Standards Test — English Language Arts (CST-ELA), 
Degrees of Reading Power (DRP), California High School Exit Exam-English Language 
Arts (CAHSEE-ELA), and student motivation between students who participated in the 
targeted SLIC intervention for two years when compared to their control counterparts 
after controlling for key covariates (e.g., pre-test score, percent with BA degree as proxy 
for socioeconomic status). 


93 


After three years of participation in the targeted SLIC program, no significant differences 
between students in the targeted SLIC intervention groups and students in the control 
groups were found for any of the selected outcomes of interest (i.e., DRP, CST-ELA, 
CAHSEE, student motivation). This held true even when examining outcomes by 
subgroups of interest (1.e., English learners, middle school students, high school students) 
and when controlling for key covariates (e.g., pre-test score, percent w/ BA). 


Using the Bloom’s adjustment (TOT analysis), there were no SLIC effect for CST for 
any of the study cohorts (one year, two years and three years of participation). However, 
for DRP, the results indicate a small SLIC (intervention) effect for one year (p<.05) and 
two years of participation (p<.05) but not for three years of participation. For one year 
and two years of participation the percentage effect of the intervention is 3.12% and 
4.46%, respectively. 


V. EVALUATION OF THE IMPLEMENTATION OF THE WHOLE-SCHOOL 
INTERVENTION: YEAR 4 


Summary of the Design 


The evaluation of the implementation drew on data from multiple sources, including 
observation (of coach meetings and cross-site and site-specific professional development 
sessions for content teachers, and classroom instruction), professional development 
materials, program documents and records of participation in professional development, 
annual interviews (with coaches, district leadership and developers), and surveys of 
teachers and students. 


Research Questions on the Implementation of the Whole-School Intervention in 
Year 4 


What was the nature and amount of professional development/support for 
teachers/coaches in Year 4? 


Professional development/support for coaches 
What was the professional development model for coaches in Year 4? 


What was the variability (amounts) of professional development /support for coaches in 
Year 4? 


What support are coaches given in understanding content-specific needs, content, and 
pedagogy? 
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Do coaches’ understandings of the SLIC model conform to the views of the program 
leadership? 


What forms of direct support (e.g., district staff presence in teacher-coach meetings) and 
indirect support do coaches have in their work with content teachers? 


What kinds/amounts of school site support do coaches have in their work with content 
teachers? 


Professional development/support for content teachers 

What types of support were provided to content teachers? 

What types of professional development were provided to content teachers? 
What amount of professional development was provided to content teachers? 
What support are teachers given in lesson planning and instruction? 


What support are content teachers given in assessing student work to determine student 
needs? 


How are teachers supported in content-specific use of SLIC? 

How are content teachers supported in classroom work with SLIC students? 
What kind and amount of support is provided by school site leadership? 
What is the proportion of teachers receiving different levels of support? 


What was the fidelity and level of implementation, and the variability of classroom 
instruction in Year 4? 


What is the Year 4 classroom model (materials, resources, strategies, assessment 
practices, etc.)? 


What proportion of teachers has access to all of the program resources that the model 
specifies? 


Fidelity of Implementation 


With respect to teaching points, does content teachers’ instruction follow the SLIC 
model? 


With respect to pedagogy, does content teachers’ instruction follow the SLIC model? 
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What types of reading materials do content teachers use with students? (e.g. grade level 
texts or teacher-provided notes). 


What kinds of reading writing tasks are students given in content classes? 

Level of Implementation 

What proportion of teachers tried SLIC instructional strategies in their classrooms? 
What did the counterfactual (for whole-school intervention) look like in Year 4? 


What were the literacy programs offered to students attending comparison schools? 
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Figure 9: Year 4 Data Sources on Implementation Linked with Research Questions: Whole-School Intervention 


Research Questions Measures/Data Sources 
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What was the nature and amount of professional development/support for teachers/coaches in Year 4? 
Professional development/support for coaches 


What was the professional X X X 
development model for 
coaches in Year 4? 


What was the variability 
(amounts) of professional 
development /support for 
coaches in Year 4? 
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What support are coaches 
given in understanding 
content-specific needs, 
content, and pedagogy? 


ani 
Do coaches’ understandings | x X X 
of the SLIC model conform 
to the views of the program 
leadership? 


What forms of direct support 
(e.g., district staff presence 
in teacher-coach meetings) 
and indirect support do 
coaches have in their work 
with content teachers? 


Figure 9 (continued): Year 4 Data Sources on Implementation Linked with Research Questions: Whole-School Intervention 
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Figure 9 (continued): Year 4 Data Sources on Implementation Linked with Research Questions: Whole-School Intervention 
Measures/Data Sources 
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Year 1 Whole-School SLIC Implementation Study 


In Year 1, progress was made in implementing the whole-school intervention in all schools, 
although the degree of classroom implementation achieved was likely low. Content-area 
teachers were exposed to the program in professional development sessions however there was 
not the sustained work with teachers that would allow them to implement the program with 
fidelity in their content-area classrooms. The main focus of Year 1 work was building “buy in” 
among content teachers and demonstrating the relevance of literacy instruction to content-area 
teaching. It was anticipated that interest in the program would grow and use of program methods 
would increase in the second year, as content-area teachers’ contact with the program expanded. 
The level of implementation may have been highest at one of the schools, although leadership 
and developers questioned the fidelity of implementation at that site. The first year was a 
planning year for the evaluation. As a result, data collection was used primarily for instrument 
development. 


The Year | Implementation Report suggests specific expectations for content-area teachers’ 
participation and other forms of training. On average these expectations were met much less 
than expectations for implementation of the targeted intervention. Details about how these 
expectations were operationalized and assessed at participating schools can be found in 
Appendix E. 


Year 2 Whole-School SLIC Implementation Study 


In Year 2, the SLIC whole-school program continued at five schools and was initiated at a 
middle school, comprehensive high school, and a “small school” in a high school complex. The 
findings in this section are based on analysis of interviews, on surveys of teachers and students, 
and observation at professional development sessions. Incentives were offered for answering the 
survey but response was optional, and it is possible that those answering were teachers more 
interested in the program. 


The district leadership and developers placed much more emphasis on the Whole-school 
program in Year 2, and progress was made at all school sites. The amount of content-area 
professional development increased, and it is likely that classroom implementation increased as 
well, although it is difficult to compare because quantitative measures were being developed in 
Year | and were not used until Year 2. While much progress was made, there were competing 
demands on program staff which diminished the progress developers intended to make in the 
Year 2. 


The manner in which professional development was provided changed in the second year, and 
the developers offered fewer large cross-site events and instead pursued more work with 
individual teachers and academic departments at schools. Generally, the two developers worked 


at a school for several days before moving on to another school, and the coaches expanded on 
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their work and initiated more contacts.'* As in the first year, the developers divided the schools 
between them, and one developer continued Year 1 work at two high schools and added a new 
high school and middle school, while the other developer continued working at three middle 
schools and added a new high school. 


The developers were unable to return to schools as frequently as they, or coaches, might have 
wished. As noted, the number of participating schools increased from five to eight in Year 2 and 
a substantial amount of the developers’, district leaderships’, and coaches’ time was absorbed by 
their work with the SLIC assessment. In addition, one coach was on leave for part of the school 
year, and the school had new teachers leading the targeted SLIC intervention classes. The coach 
at a second school was ill for part of the year and this school had teachers new to SLIC and to 
teaching. The district leadership attempted to fill in for the first coach, which afforded them less 
time to provide support for the whole-school program or the targeted intervention at other sites. 


Three of the five Year 1 coaches remained with SLIC in Year 2, and the leadership brought in 
two replacement coaches and added three new coaches to work with the new schools. Two of 
the new coaches were introduced to the program during the summer, two more were brought in 
just before the school year began, and the fifth new coach began at the end of September, after 
the start of the school year. Those beginning during the summer had considerably more training 
and exposure to the program than those arriving late. As in Year 1, the coaches were based at the 
school sites, and gathered for weekly meetings with the District leadership and SLIC developers. 


Six coaches were present for all or most of the school year, and they expanded the program’s 
contact with content-area teachers and had continuing responsibility for providing support to 
SLIC teachers. Consequently, at some sites with new SLIC teachers, there were competing 
demands for coaches’ time and attention. 


Despite these challenges, in Year 2 the whole-school program made progress in raising 
awareness of the program among content-area teachers. The teacher survey included a series of 
questions about teachers’ knowledge of SLIC and their interest in bringing it to their classrooms, 
and provided a 5-point Likert scale for response. Among non-SLIC teachers responding to the 
survey, 37% agreed or strongly agreed that they “understand what the SLIC model is,” and 
another 26% gave the middle response. On the question “I have received training that will 
enable me to bring aspects of the SLIC model to my classroom,” 33% agreed or strongly agreed, 
and another 25% gave the middle answer. 


Interest in SLIC among content teachers was higher than their confidence in their ability to 
implement the program, and 51% agreed or strongly agreed that they “...would like to bring 
aspects of the SLIC model to my classroom,” and another 28% gave the middle response. 


'S At one school the principal gave the coach broad responsibility for organizing professional development. 
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Teachers were then asked to respond to the statement that they “...have made specific plans for 
bringing the SLIC model to my classroom” and 25% of non-SLIC teachers agreed or strongly 
agreed, with another 21% giving a medium response. 


This summary of teacher survey results both over-represents and under-represents the level and 
fidelity of implementation; on one hand there is probably selection bias in the survey sample, and 
on the other hand some SLIC teachers, who are excluded from these statistics, teach English and 
Science classes in addition to SLIC, and have, according to interviews, brought SLIC methods to 
those classes. When the SLIC teachers are included, those agreeing or strongly agreeing with the 
four statements increases: from 37% to 40% for those who “understand the SLIC model”; from 
33% to 36% for those who “received training”; from 51% to 54% for those who “would like to 
bring aspects of the SLIC model” to the classrooms”; and from 25% to 29% for those who “have 
made specific plans.” In Year 3, questions will be added to directly address classroom 
implementation. 


Interviews and survey response suggest that there is highest interest in the program among 
teachers in Science, History, and English Language Arts (ELA) and somewhat lower interest 
among Mathematics teachers’”, although a few coaches have worked with Mathematics teachers, 
and the developers have worked with math departments at a few school sites. The program’s 
emphasis on expository text and on Science and History textbooks makes its use in these areas 
most obvious, while fewer models are offered of SLIC implementation in Mathematics. In Year 
2 there was greater emphasis on narrative text than in Year 1, which supported the program’s 
expansion in ELA. The developers suggested that the SLIC approach offered limited help in 
Mathematics because even though it was possible to teach students to read Mathematics 
materials, SLIC instruction was “‘often compromised because students lacked the math literacy 
they needed” such as knowledge of the operations and symbols particular to Mathematics, 
knowledge which was equally important in learning from the texts. 


Despite the progress in recruiting teachers to the program, classroom implementation was 
somewhat limited. Perhaps the strongest implementation was among SLIC teachers who teach 
content classes in addition to the targeted intervention class, because they were most sure of their 
grasp of SLIC methods. 


Interviews indicate that, as in the program’s first year, the support of site leadership was very 
important in whole-school implementation. The degree of site leadership support varied widely, 


' Responding to a statement that they “...would like to bring aspects of the SLIC model to their classrooms” there 
was agreement or strong agreement among 8 of 24 (33%) of Mathematics teachers, 23 of 39 (59%) of ELA teachers, 
15 of 19 (79%) of science teachers, and 19 of 24 (79%) of social studies teachers. And those teachers who have 
made specific plans (i.e. state that they agree or strongly agree) to bring SLIC to their classrooms include 12% of 
Mathematics teachers, 26% of ELA teachers, 53% of the 19 science teachers responding, and 43% of social studies 
teachers. 
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as did the openness of teachers to try the program. Attitudes ranged from active resistance to 
active interest. Progress was made at the sites that were part of the program in Year 1, but 
coaches at some new sites faced challenges similar to those confronting coaches in Year 1. As in 
the first year, the principal’s willingness to give the coach a leadership role made a difference, as 
did the coach’s relationships with teachers at the site. 


The importance of the coaches in promoting and implementing the program became apparent at 
two sites lacking a coach for much of the school year, and at these sites the whole-school 
program that was initiated in Year 1 lost momentum. At a few sites, SLIC teachers took on roles 
in promoting the program and providing professional development, which appeared to be an 
effective approach to implementation. 


The evaluators observed that coaches at comprehensive high schools had more work providing 
ongoing support to teachers than did teachers at “small schools,” simply because of the greater 
numbers of classes and teachers at the comprehensive schools. While it was possible for the 
coaches to reach a number of teachers, raise their interest in the program, and in some cases 
provide guidance about how to start off with implementation, it was more challenging to give 
content teachers the support that would allow them to operate confidently and implement the 
program with fidelity. Coaches began developing strategies for meeting the increasing 
professional development needs of content teachers, and these may be used in Year 3. 


Contrary to expectations, the developers and district leadership found greater interest in the 
program in the high schools than in middle schools. This was partly due to the fact that there 
was greater stability in high school staffing (with continuing principals, coaches, and SLIC 
teachers) than in middle school staffing between Year 1 and Year 2. It is also possible that there 
is greater need for the program in high schools. The California High School Exit Exam 
(CAHSEE) has raised the stakes for the students and their teachers to advance students’ literacy 
proficiency and, in addition, it is possible that middle schools are already using some of the 
methods promoted in SLIC. The teacher surveys included questions about practices such as 
teaching students to preview text features, take notes, write coherent paragraphs, and so forth, 
and the responses given by middle school teachers were consistently higher, even at schools 
where the program’s reach was, according to interviews, quite limited. It is possible that 
teaching of these skills is more common at the middle school level than the high school level. 
Whether middle school teachers are in fact doing this kind of instruction, and doing it in the way 
that SLIC leadership advocates and to the effect that leadership anticipates, is unclear. 


Year 3 Whole-School SLIC Implementation Study 


In Year 3, one of the eight schools participating in the whole-school implementation formally 
changed; although it was in the same location as a Year 2 middle school and retained some 
teachers and students, it brought the name, administration, and focus (on creative and performing 
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arts), as well as some of the students and teaching staff, of a different middle school. The 
remaining seven schools were the same. Similarly, seven of eight Year 2 program coaches 
continued in the program, while an eighth, formerly a middle-school coach, continued in a 
different support role. The proportion of content-area teachers continuing in the program also 
increased substantially in Year 3, from 37% to 76%. 


Despite this continuity, roles for supporting the whole-school implementation changed 
substantially in Year 3. In general, program coaches took on more responsibility for professional 
development and other support for the whole-school program, as program leaders took on more 
of the support for targeted intervention teachers. This change was associated with a marked 
increase in the amount of professional development and other support to whole-school 
implementation provided in Year 3. The evidence on resulting changes in classroom instruction 
is mixed, suggesting a general increase in the amount of in-class literacy practice and more 
limited implementation of the specifics of program curriculum and pedagogy. 


At the middle-school level, three of four sites provided whole-staff professional development 
sessions near the beginning of the year, but (following the trend established in Year 2) the large 
majority of PD was provided in department-specific and individualized work. Following the 
departure of one of the middle-school coaches, some of the content-area professional 
development in middle schools occurred in cross-site sessions planned by leadership and three 
middle school along with one high school coach. These cross-site PD sessions were generally led 
by program coaches. At one middle school, these sessions constituted the bulk of support for the 
content-area program in Year 3, while they supplemented site-specific work in two others. 


At the high-school level, nearly all program support to content-area teachers was site-specific 
and was provided at the department or individual level, with only one high school offering 
whole-staff PD. Models for support varied widely at high schools. At two sites, program leaders 
helped to lead and plan PD sessions. At two others, content-area support was provided almost 
entirely by the site’s coach, with some planning support provided by program leadership. The 
departmental focus of content area work differed by site, and reflected the interests of teachers at 
each site. 


The overall amounts of professional development and other program support provided to each 
content teacher increased substantially in Year 3. While in Year 2 no schools had more than 
80% of teachers in the core content areas receiving the total expected hours program support 
(more than 27 hours), in Year 3 over a third of schools did. Coach participation in professional 
development, already fairly high in Year 2, rose slightly in Year 3, with 7 of 8 coaches at the 
maximum level of support. 
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Some coaches seemed more able to overcome obstacles to implementation than others. Some 
coaches learned to proceed with considerable tact—for instance, by building individual 
relationships with teachers as a basis for working with whole departments or by running 
professional development through current teachers. Persuading department heads to lead 
program-related PD was another way of securing implementation. In one case, the site’s coach 
avoided describing her work as being connected to a specific program and offered general 
instructional support. Whether such strategies were implemented successfully seemed to depend 
in part on the coaches. While multiple sites were described as having problems with teacher 
resistance and weak support from the principal, coaches who were comfortable with building 
relationships were more successful than coaches who appeared to lack the confidence, flexibility, 
and determination to win over teachers and school leadership. 


As already noted, program developers envisioned program-related instruction in approximately 
60% of a student’s classes in a typical day. The whole-school intervention encourages teachers to 
facilitate students’ independence in using texts within their subject areas by instructing them in 
strategic use of form-specific features and structures of texts and questioning them about their 
literacy practices. By the same token, it de-emphasizes or discourages teaching practices that 
encourage students’ reliance on extra-textual resources, such as lectures, frontloading of 
vocabulary, and text-to-world connections. 


Data from surveys and from observations of content-area classes by the evaluation team suggest 
varying levels of fidelity to these principles. Overall, surveys suggest frequent implementation of 
literacy instruction and practice in content-area classes. When intervention school students were 
asked to indicate frequency of reading in each of the four core content areas, 45% of responses 
were “always” or “often.” 50% of teachers reported having students read independently in their 
classes 6 or more times in the previous semester. Many more specific forms of literacy practice 
and instruction received similarly high ratings from both teachers and students. However, there 
was little evidence on either student or teacher surveys of changes in practice between Year 2 
and Year 3. 


During classroom observations, instruction and practice in reading or writing were widespread, 
occurring for more than 20 percent of class time in 18 out of 20 observations in intervention- 

; : : : «20 
school classes in English, science, and social studies. 


°° These observations occurred in a sample of classes of teachers from 4 schools reported to have relatively high 
implementation among the 8 intervention schools. Teachers were both randomly sampled (9 of 20) and 
recommended by program staff as exemplars of program implementation (11 of 20). Observations were scheduled in 
advance, and despite our best efforts to persuade teachers that a typical lesson was desired and that there were no 
risks associated with our observations, at least one teacher believed she needed to display her literacy instruction. As 
such, there could be a positive bias to our observations. Yet it should also be noted that the observers not 
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The foregoing suggests that while students may not yet be exposed to the intervention approach 
in 60% of their classes, most students are likely encountering something like the program 
approach fairly often in a range of classes. It should be noted, however, that these positive 
indications do not necessarily reflect the program’s impact alone. As was also reported in Year 2, 
there is evidence that more than a few teachers feel they were making use of program-specific 
literacy strategies prior to their involvement in the program. 


Year 4 Whole-School SLIC Implementation Study 


There was relative continuity (81%) in Year 4 in the staffing of content area teaching positions, 
and greater stability of staffing in middle schools (averaging 85% and ranging from 74% to 97%) 
than in high schools (average 74%, with a range of 67% to 80%)*'. Stability increased the 
likelihood that Year 4 teachers had participated in SLIC professional development in prior years 
and increased the cumulative influence of Striving Readers at the schools. At the same time, the 
number of on-site coaches fell sharply when five coaches left the program at the end of Year 3. 
Three schools had no coach and two schools shared a coach. The leadership continued to focus 
on the targeted intervention program as they had in Year 3, while content area PD was the 
responsibility of the remaining coaches. The cross-site team of middle school teachers organized 
in Year 3 was dissolved and, with the exception of a coach working at two middle schools, 
coaches worked exclusively at their school sites. (See Table 29.) 


Table29. New and Continuing Content-Area Teachers in the Whole-School SLIC Program, 
Years 1-4 


Year 1 Year 2 Year 3 Year 4 
% (#) of Teachers New to 
SLIC 100% (104) 63% (111) 24% (45) 19% (43) 
% (#) of Teachers Continuing 
from Previous Years 0% (0) 37% (66) 76% (139) 81% (181) 


infrequently rated randomly selected teachers higher on fidelity-rated measures than exemplar teachers, sometimes 
in comparison to a teacher in the same school and subject area. 
*! Tn Year 4 there were 116 middle school and 108 high school content area teachers at the intervention schools. 
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Partly as a result of these changes and in conformance with the leadership’s view that the 
program should adapt to the needs and resources of the site, there was little consistency in Year 4 
in the content area work across school sites. There was expansion at some schools and decline at 
others. At one school with no coach, the SLIC teachers provided some content area professional 
development. At another school with no coach, the departing coach organized department 
chairmen to provide some continuity in carrying out SLIC professional development within their 
departments. While professional development continued at this school, there was no check on the 
fidelity of professional development to the program model. At another school, a coach who had 
moved to another school returned to provide a few Saturday sessions for interested content 
teachers but the already somewhat low profile of the program at the school slipped further. At 
another school a SLIC teacher who was well integrated at the school moved to a coaching 
position and this former teacher provided professional development support to content teachers. 
And, as mentioned, one coach worked at two school sites. The three coaches who remained at 
their schools were by now well integrated and accepted, and the sites had continuing professional 
development and content area implementation. At other sites, the programs’ profile depended 
partly on the breadth and depth of the program’s influence in prior years, principals’ support and 
the continuity of the school’s content teaching staff. 


The departmental focus of the work differed across the eight intervention schools. At the schools 
overall, the greatest number of hours of professional development were provided to English 
Language Arts teachers (averaging 37) and science teachers (23 hours). All science teachers 
participated in professional development at six of the eight schools while there were no science 
department participants at the other two schools, and all math teachers participated at six schools. 
English departments, in contrast, had full department attendance at only three schools. One 
school had fairly high participation in the math department, both in the average hours of 
professional development and proportion of the department participating, but no coverage in 
science. One school had high participation across subjects and low average hours of 
participation. There was broader coverage at the middle schools than at high schools and 
narrower or more focused participation at high schools. 


In general, content teachers participated in the program if they were interested in doing so. 
However the principals also had a significant role in encouraging or requiring participation, and 
the relationship between coach and principal and the coach’s ability to convince the principal of 
the program’s value seemed to figure strongly into whether the principal was willing to require 
participation. At the same time, the principal’s security in their position was crucial, as was the 
teaching staff’s length of tenure and sense of school ownership and the relationship between the 
principal and influential teachers. The absence of a coach resulted in substantially lower levels 
of professional development at one site and somewhat lower levels at another. (See Table 30.) 
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Table 30. Implementation of Whole-School SLIC PD/Support, Years 1-4 


Content-Teacher Professional Development/ PD Support* (% (#)) 


Year 1 Year 2 Year 3 Year 4 
Low 80% (4) 63% (5) 13% (1) 25% (2) 
Medium 20% (1) 38% (3) 50% (4) 63% (5) 
High -- -- 38% (3) 13% (1) 


Coach PD Participation 


Year 1 Year 2 Year 3 Year 4 
Low es = we 38% (3 sites**) 
Medium 40% (2) 25% (2) 13% (1) 50% (4 sites) 
High 60% (3) 75% (6) 88% (7) 13% (1 site) 


* Ratings for content-teacher participation in professional development are based on the 
percentages of teachers receiving the expected amount of program-related support (27+ 
hours) at each site for a given year: Low = < 20%, Medium = 20%-60%, High = > 60%. 
** Three sites had no coach. 


The levels of PD support declined in Year 4 from Year 3. However, there was considerable 
variation across the eight schools, with high levels of PD at schools with coaching support and 
fairly low levels at the other schools. 


Ratings for fidelity of classroom instruction depend on aggregate measures of teacher and 
student responses to the survey questions. Details on these measures can be found in Appendix 
E. Year 4 saw a decline in the number of schools achieving medium levels of classroom 
implementation and none of the schools achieved a high overall implementation adequacy score. 
(See Table 31.) More specifically, four middle schools and three high schools achieved a 
medium level of implementation with respect to the average adequacy score. (See Table 32.) 
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Table 31. Level of Implementation of Whole-School SLIC (Schools=8), Years 2-4 


Fidelity 


Year 2 
Low 
Medium 


High 


Year 3 
Low 
Medium 


High 


Year 4 
Low 
Medium 


High 


Content-Teacher 


Professional 

Development/ 

PD Support* (% Coach PD Classroom Average 
(#)) Participation Model Adequacy Score 
63% (5) -- 25% (2) -- 

38% (3) 25% (2) 75% (6) 100% (8) 
-- 75% (6) =o =- 

13% (1) = 25% (2) ce 

50% (4) 13% (1) 63% (5) 75% (6) 
38% (3) 88% (7) 13% (1) 25% (2) 
25% (2) 38% (3) 38% (3) 13% (1) 
63% (5) 50% (4) 50% (4) 88% (7) 
13% (1) 13% (1) 13% (1) zs 
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Table 32. Level of Implementation of Whole-School SLIC (Schools=8), Year 4 (2009-10), by 
School Level 


Fidelity Content- Coach PD Classroom Average 
Teacher Participation Model Adequacy 
Professional Score 
Development/ 
Support (% (#)) 

Middle Schools 

Low 1 1 (no coach) 2 -- 

Medium 2 3 2 4 

High 1 -- -- -- 


High Schools 


Low 1 2 (no coach) 1 1 
Medium 3 1 2 3 
High -- i 1 -- 


Classroom Implementation 


An important question is whether the program has affected the classroom practice of content area 
teachers. An open-ended question on the teacher survey asks if teachers’ instructional practice 
has changed due to SLIC, and 66% of teachers gave examples of the way it had changed their 
practice, while 24% said their practice had not changed, and 10% did not know enough about the 
program. Among the positive comments, teachers mentioned that reading and writing have 
become a greater focus in their classes, that they were more aware of the need to differentiate 
instruction for students, that they include more instruction in the use of text features and notes, 
and they do more careful planning and thinking about the background knowledge students need. 
Negative comments were most often simply “no” though some said they already followed 
similar practices. 


Teacher and student surveys have somewhat dissimilar results with respect to literacy practices 
in content classes at intervention and comparison schools. The teacher survey shows the 
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intervention schools lower than comparison schools in Year 2 and higher in Year 3 and virtually 
the same in Year 4. Additionally, the intervention schools show no increase in the mean 
response to these questions from Year 2 to Year 4. Student surveys suggest more literacy 
activities in classrooms of comparison schools than intervention schools in all three years. 


With respect to the middle school/high school contrast, teachers report higher literacy practices 
at intervention middle schools than intervention high schools while middle/high school results 
are mixed in comparison schools. On the student surveys, the middle school/ high school contrast 
suggests more common content area literacy practice in high schools in both the intervention and 
comparison samples. (See Tables 33 and 34.) 


Table 33. Mean Frequency of Literacy Activities in Class by Content Area Teachers per 
Semester, by Year 


Year 2 Year 3 Year 4 
Mean SD Mean SD Mean SD 
Intervention Schools 2.33 0.93 2.36 0.96 2.29 0.93 


Middle Schools 2.59 0.94 2.45 0.99 2 0.96 

High Schools 2.08 0.86 pe ae 0.92. 2.24 0.92 

Comparison Schools 2.38 0.92 221 1.00 221 0.97 
Middle Schools 2.31 0.95 2.33 1.00 Zao. 0.97 

High Schools 2.53 0.80 2.16 0.98 2.18 0.95 


Note. The question asks how many times the teacher did each of the following in the previous semester using a five- 
point scale representing 0 times, 1-2 times, 3-5 times, 6-10 times, 11+ times. Items (15a-n) are: Have students read 
independently in class; have students preview text features and text organization in a reading assignment; teach key 
vocabulary prior to teaching — scores reversed; teach students how to understand unfamiliar vocabulary using 
context or morphology; teach or model how to find the main idea in a text; have students find the main idea(s) in 
their reading; teach students how texts in your discipline are structured, including textbooks and supplemental 
sources; model note-taking based on reading; teach or model how to preview the text and use the structure to set up 
notes; have students take notes independently, based on in-class reading; teach or model how to locate information 
in texts; explicitly teach how writers in your discipline state and support main ideas; teach students to cross-check 
their understanding as they read; teach or model how to critically assess an author’s arguments or use of evidence. 
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Table 34. Content Teachers’ Implementation of Whole-School SLIC Model, by Year 


I have made specific plans for bringing 


the SLIC model to my classroom. 


I have tried the SLIC model in my 
classroom. 


I use the SLIC instructional model in 
my classroom a few times a week. 


Year 2 
Mean SD 
1.72 1.42 


Year 3 
Mean SD 
2.06 1.28 
2.24 1.47 
1.54 1.33 


Year 4 
Mean SD 
1.78 1.35 
2.01 1.39 
1.43 1.36 


Student response to a set of questions enquiring about literacy practices in English, math, 
science, and social studies classes show no change at intervention schools across years, and show 
slightly lower response at intervention than at comparison schools. (See Table 35.) 


Table 29. Students’ Literacy Practices by Content Class, by Year 


Mean 


Intervention Schools 3.28 
Middle Schools 3.21 

High Schools 3.34 

Comparison Schools 3.38 
Middle Schools 3.27 

High Schools 3.47 


Year 2 


SD 

0.95 
0.91 
0.97 
0.85 
0.81 
0.87 


Mean 


3.28 
See 
3.36 
339 
3.35 
3.41 


Year 3 


SD 

0.72 
0.70 
O73 
0.70 
0.68 
0.71 


Mean 


32) 
3.18 
3:33 
3.49 
3.45 
5.52 


Year 4 


SD 

0.72 
0.65 
0.77 
0.68 
0.66 
0.69 


Note. Student survey questions 6-9, a-f: How often do you read in class in your English Language Arts 
class; how often do you write in class in your English Language Arts class; how often do you 
preview text features before reading in your English Language Arts class; how often do you 
underline important details in your English Language Arts class; how often do you take notes when 
reading in your English Language Arts class; how often do you use notes to help with an assignment 
in your English Language Arts class. Same questions were asked for history, science, and math. 


Questions on five-point scale from “never” to “always.” 
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Therefore, in general, teacher surveys suggest a decline in implementation at intervention 
schools while student surveys indicate little change across years. The teacher and student 
surveys are somewhat contradictory with respect to the contrast between literacy practices at 
intervention and comparison schools: teacher surveys suggest higher comparison school 
implementation in Year 2, slightly higher intervention school implementation in Year 3, and 
virtually the same level of literacy practice in Year 4. Student surveys suggest higher comparison 
school implementation in all years. 


Classroom observations provide another source of information about implementation of the 
whole-school program. Observations of English, science and social studies classes were 
conducted in twenty-three intervention and thirteen comparison classrooms, with observations 
taking place in winter and spring of the 2009-10 school year. Here, the results indicate that there 
are differences between intervention and comparison schools, with the higher levels of literacy 
practice in content area classrooms at intervention schools. It is possible that the presence of 
observers had in influence on instruction, however by Year 4 however the presence of observers 
drew less attention. 


Intervention school content area classes spent more time in literacy activities than did 
comparison school classrooms. During observations, a note was made every five minutes 
regarding the literacy or non-literacy focus of instruction, and this provides a strong measure of 
the amount of class time spent on literacy. An average of 53% of class time was spent on literacy 
in the comparison school classrooms while an average of 65% of class time was spent on literacy 
in intervention school content area classes. The observers noted more literacy instruction in 
middle schools than high schools, with 67% of class time at intervention middle schools and 
62% at high schools; 64% of class time involved literacy instruction at comparison middle 
schools and 41% of class time at comparison high schools. 


Summary of observation data appear in Table 36. The scale shows the range of ratings used for 
each item, and the observer agreement ratio indicates the proportion of observations on which 
observers agreed in their ratings. Building literacy knowledge is a summary rating, and 
addresses the observers’ sense of the strength of the literacy instruction. Intervention school 
classrooms had higher ratings on literacy instruction, on the amount and quality of literacy 
practice, on the amount of instruction encouraging metacognition, on the academic rigor of the 
class, and on the kind of questioning, i.e. whether teachers used process questions (e.g. how did 
you figure that out, where did you find that information), a practice advocated by the program, or 
instead asked questions about the content of the lesson. Intervention schools had lower mean 
ratings on achieving independent work, which refers to the proportion of on task behavior (a 
rating of three meant that less than 20% of behavior was off task). Intervention and comparison 
schools received virtually the same rating on classroom environment, which refers to behavior 
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and classroom management. And the distribution of questioning (the proportion of students 


included in questioning) was higher in comparison schools. 


Based on these observational ratings, there was more literacy-based activity and more SLIC- 
related literacy activity in intervention schools in Year 4 than at comparison schools. 


Table 30. Observer Ratings of Literacy Practices in Intervention and Comparison Schools 


Scale 

Building Literacy 1-5 
Knowledge 

Literacy Instruction 1-3 
Literacy Practice 1-3 
Achieving Independent 1-3 
Work 

Classroom Environment 1-3 
Metacognition 1-3 
Academic Rigor 1-3 


Distribution of Questioning —1-3 


Kind of Questioning 1-3 


Observer 
Agreement 
Ratio 


97% 
90% 
90% 
87% 
771% 
97% 
771% 
80% 


84% 


Intervention 
Classrooms 
(N=23) 


Ze, 
1.98 
2.61 


Dede 


Comparison 
Classrooms 
(N=13) 


De2y 


1.65 


2.54 


2.69 


2dT 


1.00 


1.31 


1.85 


1.31 


Middle School/High School Differences in Fidelity of Implementation 


In Year 3, middle schools and high schools had substantially different approaches to providing 


PD and other support. These differences dissolved in Year 4 as the middle school cross-site 
coaching teams were discontinued. It should be noted that, based on the small sample of 
schools, differences between high schools and middle schools, both in amounts of support 


provided and in classroom implementation, could easily be caused by factors unrelated to school 


level in itself. 
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Classroom implementation of the whole-school program displayed middle/high school variation 
in Year 3; middle schools were equally divided between medium and low ratings, while no high 
schools were rated low and one was rated high. 


In Year 4 teacher surveys suggest higher implementation at the middle schools, while student 
surveys indicate higher implementation at high schools. The observation data show a slightly 
greater level of implementation at the middle schools on average. While middle schools had 
higher average observation ratings, one middle and one high school had the highest average 
ratings and were rated higher on five or more of the observed literacy practices listed in the table 
above. These two schools were also consistently higher on the teacher and student survey results 
for Year 4 and the overall fidelity measure. Another high school received high marks on four 
observation ratings and a middle school scored relatively high on two ratings. The high schools 
with higher ratings were both small schools and had coaches, while the large comprehensive 
high schools (which had no coaches) had low to medium ratings for whole school 
implementation in Year 4. 


Changes in Implementation from Year 1 to Year 4 


The teacher and student survey results on content area classroom implementation for the sample 
overall are fairly stable from Year 3 to Year 4. While teacher and student surveys suggest that 
two schools had their strongest results in Year 4, four or five schools had strongest results in 
Year 3. The teacher and student surveys were not consistent with respect to which schools had 
their highest ratings in a particular year, although two schools, a middle and a high school, had 
similarly high results on the Year 4 teacher and student surveys and observations and generally 
showed progress from Year 2 to 3 to 4. On the teacher surveys, the highest average year for 
Intervention middle and high schools was Year 3. Mean result from student surveys show little 
average change in content area literacy practice across three years at either Intervention or 
Comparison schools. 


The amount of professional development provided to content-area teachers increased from Year 
1 to Year 2 and again from Year 2 to Year 3 and dropped in Year 4 at some schools and 
increased at others. Over the course of the four years, more content teachers were reached by the 
program and involved in sustained work with program personnel. In Year 2, surveys suggested 
that there was more interest in the program than developers and coaches were able to support. In 
Year 3, role structures changed substantially so that, in general, program coaches had less 
responsibility for intervention teachers than in Year 2 and more responsibility for content-area 
PD. This change in role structure looked different in middle schools and high schools, with 
middle-school teachers being reached through more formal PD sessions and less individual 
work, and high school teachers at three sites being reached more on an individual basis. In Year 
4, sites with coaches had more professional development and these sites were more often middle 
schools or small high schools. 
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In Years 1-3, science teachers showed the most interest in participating in the program. By Year 
3, English teachers showed the next most interest, although English teachers’ pre-existing ideas, 
some of which are inconsistent with the program model, have also posed a challenge to 
implementation. Finally, in Year 3, math teachers became an unexpected focus of coaches’ 
content-area work, and many math teachers showed strong interest in working with the program. 
In Year 4, English teachers offered the highest response to the statement that they would like to 
bring aspects of the SLIC model to their classroom, although science teachers more commonly 
indicated that they had made specific plans and had tried the SLIC model in their classes. Yet 
English and social studies teachers were slightly more likely to state that they use the SLIC 
model in their classes a few times a week. The lowest response to all these questions came from 
math teachers, although some coaches found the greatest and most unexpected interest expressed 
by particular math teachers. 


With respect to school level, a gap between high school teachers and middle-school teachers in 
interest in the program narrowed in Year 3 and, by Year 4, the differences seemed more specific 
to the circumstances at the schools. The observation data show slightly higher average 
implementation at middle schools than at high schools in Year 4, and mean response to teacher 
survey questions on SLIC practice suggest higher middle school implementation in all three 
years. In contrast, student surveys indicate higher implementation at high schools in all three 
years, as do the set of teacher survey questions that asks teachers whether they have tried and 
frequently use SLIC. In short, the results are mixed. Two schools, a middle and a high school, 
show higher levels of professional development and classroom implementation across all 
measures, and progress from Year 2 through Year 4. And one high school has low ratings across 
all measures. 


Some challenges to implementation have continued across all four years—in particular, issues 
related to inadequate leadership support at some schools and staff resistance in some schools. 
While there was substantial change in schools and program staff between Years 1 and 2, and 
substantial demands on the time of district leadership, developers, and coaches from 
development of the SLIC assessment, both of these factors were less significant in Year 3 or 4. 
With the shift of coaches’ work toward the whole-school intervention, it appears that meeting 
initially established expectations for content-area teachers was more feasible in Year 3 but was 
threatened by the loss of coaches in Year 4. At the same time, Year 3 and Year 4 saw an increase 
in the proportion of content-area teachers with prior program experience. School leadership’s 
willingness to support the program and the inclusion of well-liked teachers as professional 
development leaders had a positive influence at a couple of schools. 
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Implications for Impact Analysis of Whole-School Intervention 


Although we focus primarily here on outcomes from only one year of the whole-school program 
(Year 4), earlier and later years of data are considered. This raises the question of how changes 
in program implementation may affect impact analyses, i.e., based on the pooling of data across 
cohorts. In general, it seems that the program’s core principles have remained consistent across 
years, although this is a tentative claim, given the size and diversity of the whole-school 
program. At the same time, the quantity of support per teacher provided through the program 
clearly increased through Year 3 while the levels of support were more variable in Year 4. With 
the ongoing and more individualized training of teachers, the quality of this support may have 
increased. As such, we might expect that the greatest program effects will be found in its later 
years and that pooling across all study years may tend to dilute the program’s support. 


However, there is currently not sufficient evidence to conclude that increased support has led to 
increased fidelity of classroom instruction program-wide, which might lead us to expect different 
program outcomes across time. Based on teachers’ self-reports, the current level of literacy 
instruction seems to be high, but there does not seem to be a substantial increase from Year 2 or 
3 of the program. 


Observations and survey data also suggest that, underneath this generally high level of literacy 
instruction, there is still considerable variation among content-area teachers in the level and 
manner of implementation. While our data do not allow us to determine the causes of this 
variation, a few hypotheses are worth considering. First, teachers’ subject matter is likely to be a 
factor in teachers’ willingness to take on the program model in their classrooms, with science 
teachers in particular being ready to see the benefits of teaching students to recognize and use the 
text features that are prominent and crucial in science textbooks. Second, site leadership may be 
an important source of variation—at some sites, principals have apparently conveyed to teachers 
the urgency of implementing the SLIC program, while others have been less forceful. Third, 
school size may be another important school-level factor, one that is likely to mediate the 
influence of site leadership as well as the likelihood that coaches can provide adequate support to 
a sufficient number of teachers. That is, at the largest sites, it may not be feasible for the program 
to be implemented effectively without a larger program staff to work and communicate with 
teachers. 


Most optimistically, one might expect that increased program support has contributed to an 
increased amount of literacy practice in classrooms and that, in combination, these would lead to 
a positive impact on student outcomes. In the absence of an effect, however, it would not be 
difficult to point to multiple obstacles and limitations—at some sites, limited school leadership 
support; at some sites and in some content areas, resistance to program implementation; or 
questionable fidelity to the program model in some of the PD and other support, which could, in 
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turn, be tied to role structures, school-level change, or limitations in the presence of coaches or 
program leadership. 


VI. EVALUATION OF THE IMPACTS OF THE WHOLE-SCHOOL INTERVENTION: 
YEAR 4 


Study Design 


The whole-school intervention was evaluated using a quasi-experimental design comparing the 
achievement of students enrolled in participating eight intervention and eight comparison 
schools. The intervention schools are middle school/high school clusters, whereby middle 
school students feed into a particular high school. For the comparison schools, there are two 
intact comprehensive high schools and their main feeder schools. The third feeder group consists 
of a middle school that feeds into a small school on a single high school campus (the small 
schools were created from the large comprehensive school as part of the district’s high school 
reform initiative). The remaining comparison schools are not part of intact cluster groups. These 
include a small high school in a different small schools complex and a comprehensive middle 
school. 


Sampling Plan 


By design, all students in the eight intervention schools (grades 6-8 in middle schools, grades 9- 
12 in high schools) were eligible for the whole-school SLIC program. Thus, no sampling of 
students was conducted. All students matriculating at the 16 intervention and comparison schools 
were included in the impact analyses for the whole-school analyses. 


Sample Size and Power 


For the whole-school SLIC intervention analyses, assuming a fixed effects model, we have 
sufficient power (80%+) to detect small to medium effect sizes of .14 to .51, depending on the 
outcome measure (e.g., CST-ELA, DRP, CAHSEE, Motivation). The minimum detectable 
effects (MDE) include adjustments for baseline covariates (see Table 37). 
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Table 37. Minimum Detectable Effects with 80% Power for Whole-School Analyses 


Full English 

Sample Learners Middle School High School 
CST 0.18 0.17 0.10 0.36 
DRP 0.25 0.24 0.42 0:23 
CAHSEE 0.49 0.51 --- --- 
Motivation Y4 0.14 0.24 0.17 0.17 


Notes: Minimum detectable effects are calculated in two stages. First, we multiply the standard 
error of the impact estimate by the appropriate Z score (2.8). Second, we convert the result into a 
standardized effect size by dividing by the standard deviation of the control group. 


Description of the Counterfactual 


Students most similar to eligible Striving Readers in intervention schools (students performing 2 
years or more below grade level in reading) participate in Literacy Advancement Academy 
(LAA) in comparison schools. In Year 3 (2008-09), four of the eight comparison schools (two 
middle and two high schools) offered LAA classes targeting primarily grades 7-10. One high 
school offered it for 9" graders only. Participation in the LAA can also take the place of an 
elective course. For example, at one school, students participate in the Literacy Advancement 
Academy for 3 hours a week in lieu of a foreign language. In Year 4 (2009-10), two of the eight 
comparison schools (one middle and one high school) offered classes titled Literacy 
Advancement Academies. 


Data Collection Plan 


The same instruments are used in the targeted and whole-school SLIC evaluations. For 
information about instruments used in the whole-school evaluation, please refer to ‘Data 
Collection Plan’ section in ‘Evaluation of the Impacts of the Targeted Intervention.’ 


Schedule of Data Collection 


The same schedule of data collection applies to both the targeted and whole-school SLIC 
evaluations. For information about the schedule of data collection, please refer to “Schedule of 
Data Collection’ section in ‘Evaluation of the Impacts of the Targeted Intervention.’ 


Summary of Analytic Approach to the Impact Analysis 
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As with the targeted SLIC evaluation, the whole-school SLIC evaluation analyzed student 
outcomes cross-sectionally using two-level hierarchical models (students nested within 16 
schools). Students in 8 intervention schools were compared with students in 8 comparison 
schools on selected outcome indicators (e.g., CST-ELA, DRP, CAHSEE-ELA, self-reported 
level of motivation), controlling for key confounding variables (e.g., percent w/ BA, student 
gender, English learner status, grade level, pre-test scores on outcome variable). The whole- 
school evaluation included 6" thru 12" graders. All covariates had less than 10% missing data 
except for parent education (over 40% had “unknown” code). As such, percent with Bachelor’s 
degree by zip code was used as a proxy for socioeconomic status. Additionally, subgroup 
analyses were conducted for English learners, middle school students, and high school students 
on the same selected outcome indicators controlling for the confounding variables outlined 
above. 


Additional analyses were conducted to examine SLIC student performance based on specific 
SDUSD Performance Goals, separate from the Intent-to-Treat model. These are presented in 
Appendix C. 


Description of Sample 


Although the 16 schools participating in the whole school evaluation are in the same school 
district, they are not chosen at random. The eight intervention schools are comprised of four 
middle/high school clusters in a geographically compact area. They serve a largely socio- 
economically disadvantaged student body (74-100% receive free or reduced price lunch) with 
high percentages of English learners (12-44%). The comparison schools include two 
middle/school clusters in a wider geographic area with fewer socioeconomic extremes (29-87% 
receive free or reduced price lunch; 10-43% are designated as English learners). Both types of 
schools serve comparable percentages of students with disabilities (7-19% for intervention 
schools, 8-20% for comparison schools). (See Table 38.) 


The intervention schools matriculate fewer students (2 comprehensive, 2 small schools) than the 
comparison schools (3 comprehensive, | small school) although their enrollments have remained 
generally stable over time. More specifically, in the second year of the study (2007-08), there 
were 16,627 students (6,498 students in intervention schools, 10,129 students in control schools) 
enrolled in the 16 study schools. In the third year of the study (2008-09), there were 16,895 
students (6,809 students in intervention schools, 10,086 students in control schools) in the study. 
In the fourth year of the study (2009-10), there were 16,310 students (6,647 students in 
intervention schools, 9,663 students in control schools). 


Interestingly, since the start of the Striving Readers study in 2006-07, the percentages of students 
qualifying as socio-economically disadvantaged and who are designated as English learners have 
increased annually overall. In addition, most of the intervention and comparison schools 
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experienced growth in their California Standards Test scores over time, much of this was a 
District-wide phenomenon (San Diego Unified School District, School Accountability Report 
Cards, 2005-06, 2006-07, 2007-08, 2008-09). 


On a number of measures, students in intervention schools differ from students in comparison 
schools. All differences discussed below are highly significant based on relevant tests of 
significance, except where noted. 


There was considerable racial-ethnic variation across sites, as seen in Table 44 below. Hispanic 
students make up the largest share of students in both groups and years, but are more represented 
in intervention schools than in comparison schools. The second most common ethnicity in 
intervention schools is White students and, in comparison schools, Filipino students. The 
proportion of male students was slightly higher at comparison schools than at intervention 
schools 49% versus 54% in Year 4 (See Appendix A, Table A6). 


Table 31. Race/Ethnicity by Study Group, 2009-10 


Comparison Intervention 


% (N=9,663) (N=6.647) 
African American 14.8 12.4 
Alaskan Ind./Nat. Am. 0.4 0.7 
Asian 3.1 2 
Filipino 254 See 
Hispanic 34.0 47.4 
Indochinese ad 6.6 
Pacific Islander 1.5 0.9 
White jae 26.4 


In all program years, there were also a slightly higher proportion of English learners at 
intervention schools. In Year 4, the whole-school population in intervention schools was 25% in 
intervention schools and 31% in comparison schools. Although these percentages have been 
fairly stable across program years, they range widely across schools—from as high as 44% and 
as low as 15% in intervention schools, and as high as 49% to as low as 18% in comparison 
schools (see Appendix A, Table A6). 


Percent with Bachelor’s degree (using zip code data) was significantly different between 
intervention schools and comparison schools, with comparison schools having higher average 
education (18% versus 16%) (see Appendix A, Table A6). 


With respect to the main reading tests that qualify students for the SLIC targeted intervention 
class, the whole-school populations of intervention and comparison schools are similar. On the 
English Language Arts portion of the California Standards Test (CST-ELA), comparison school 
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students averaged 346 and intervention school students averaged 343 on the Year 4 pre-test, 
administered in the Spring of 2009 (see Appendix A, Table A6). Both scores are in the “basic” 
range. On the Degrees of Reading Power (DRP) pre-test (Spring 2009), intervention school 
students slightly outperform comparison school students at the highest proficiency level, as 
shown in Table 45 but the overall distributions are quite similar, with only a marginally 
significant difference (p = 0.023). (See Table 39.) 


Table 32. Degrees of Reading Power (DRP) Test by Study Group 


Comparison (N = 4,906) Intervention (N = 3,773) 


At or above grade level 48% 50% 
Close to grade level 14% 14% 
Below grade level 14% 14% 
Far below grade level 24% 22% 


Finally, a substantially higher share of students have special education placements in 
intervention schools (16%) than in comparison schools (11%). By far the largest qualifying 
disability category in both groups is specific learning disabilities - 8% in intervention schools 
and 7% in comparison schools. 


Impacts on Whole-School SLIC Students — One Year Participation 


This section displays the student outcome results for one year participation (in Year 4) in the 
whole-school SLIC program for all students in the 16 study schools. The analyses were first 
conducted for the full sample of treatment and control students based on one year of participation 
in 2009-10. Additional analyses were conducted to examine student performance by subgroups 
of English learners only, middle school students, and high school students. 


Student reading skills were measured using assessments already in place by the San Diego 
Unified School District (SGSDUSD). Analyses were run to determine what difference, if any, the 
whole-school SLIC program had on students’ performance on two reading assessments, in 
comparison to students in the comparison schools: Degrees of Reading Power (DRP) and 
California Standards Test — English Language Arts (CST-ELA). Analyses were not run with 
Advanced Placement passage as an outcome measure due to a small sample size. 


Other student academic outcomes were measured to assess the broader impact the SLIC program 
may have had on student performance. First, performance on the California High School Exit 
Examination (CAHSEE) was used as an outcome measure. Finally, student motivation was 
measured using a scale constructed from student responses to twelve survey questions (see 
Appendix B). All items ranged from “0” (not true) to “4” (very true), so that the resulting scale 
ranged from 0 (not at all motivated) to 48 (very motivated). The scale was highly reliable 
(Cronbach’s Alpha=.873), suggesting that the questions measured a single, unidimensional latent 
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variable. Two motivation outcome scales were created — one for Y3 and another for Y4. Separate 
regression analyses were conducted on each motivation outcome using the previous year’s 
motivation scale as a covariate. 


Examination of the mean pre-test scores for both the whole-school SLIC students (intervention 
schools) and control students (comparison schools) suggest that both groups were fairly 
comparable at baseline and on the CST-ELA and DRP post-tests (following one year of 
participation in whole-school SLIC) (see Table 40). 


Table 40. Mean Assessment Scores for Whole-School SLIC and Comparison School 
Students (One Year Participation — Year 4) 


Comparison Schools Intervention Schools Total 
Pre Post Pre Post Pre Post 


352.27 353.13 349.40 351.74 351.07 BRP ae) 
CST-ELA = (53.27) (57.06) (51.56) (55.20) (52.58) (56.29) 
n=6823 n=6823 N=4915 n=4915 n=11738 = n=11738 


DRP 33.19 57.63 53.87 57.69 53.80 57.66 
(15.03) (16.06) (14.80) (15.34) (14.93) (15.74) 
n=2859 n=2859 n=2234 n=2234 n=5093 n=5093 


Note: Each cell includes the mean, standard deviation and sample size. CST-ELA=California Standards 
Test-English Language Arts; DRP=Degrees of Reading Power. 


Table 41 displays raw and regression-adjusted means for treatment and control groups, the 
differences in the regression adjusted means (treatment minus control), the standardized effect 
size (SES) of the treatment, and probabilities for the null hypothesis that the treatment and 
control groups were equal. The SES was calculated for each regression model by dividing the 
difference in regression-adjusted means by the standard deviation of the control group. 


No significant differences between treatment and control groups were found for CST-ELA 
scores, DRP scores, CAHSEE passage, or student motivation. More detailed results for each 
regression analysis are given in Appendix D. 
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Table 33. Estimated Effect of Whole-School SLIC Intervention on Student Outcomes for 
Intervention and Comparison School Students (Year 4) 


Raw Means Regression-Adjusted Diff Comparison SES _ p- 
Means Group SD value 
Comp. Interv. Comp. Interv. 
CST 353.13 = 351.74 3o2733 355.11 2.78 57.06 0.05 0.46 
DRP 57.63 57.69 57.81 58.15 0.34 16.06 0.02 0.81 
CAHSEE 374.74 374.72 372.51 STT20 4.69 38.40 0.12 0.49 
Motiv. (Y4) 35.70 34.48 35.47 35.06 -0.41 7.80 -0.05 0.31 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted means 


by the standard deviation of the control group. CST-ELA=California Standards Test-English Language Arts; 


DRP=Degrees of Reading Power, CAHSEE=California High School Exit Exam. 


Subgroup Analyses 


Additional HLM analyses were conducted to determine whether SLIC classes had an impact on 
selected subgroups, specifically English learner students, middle school students, and high 
school students. These models included the same dependent variables and covariates as the 
analyses conducted on the whole sample, but the sample was only the subset of students. 


English Language Learners 


The results of these analyses, displayed in Table 42, reveal no significant differences in the way 
targeted SLIC classes affected English learner students in comparison to the rest of the sample. 
Participation in the whole-school SLIC intervention had no significant effect on any measured 
student outcomes for English Learners. The English Learner subset analyses should be 
interpreted with caution, since statistical power is significantly reduced given the smaller sample 
of cases. 


Middle and High School Students 


Finally, analyses were conducted to detect differences in the causal effect of SLIC between 
middle school and high school samples. Differences between treatment and control groups 
remain statistically indistinguishable from zero for both middle school and high school samples 
with regards to achievement on the CST and DRP. For student motivation, however, for middle 
school students the comparison schools had slightly higher mean adjusted scores (35.70) when 
compared to the intervention middle school students (34.50) significant at p<.01. Tables 43 and 
44, show the estimated effect of SLIC on student outcomes for middle school and high school 
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students, respectively. It should also be noted that inclusion of the student motivation scale as a 
control in the other above regression equations did not significantly affect findings. 


Table 34. Estimated Effect of Whole-School SLIC Intervention on Student Outcomes for 
English Learners (One Year Participation — Year 4) 


Raw Means Regression-Adjusted Diff Comparison SES _ p- 
Means Group SD value 
Comp. Interv. Comp. Interv. 
CST 341.41 324.11 335.07 335.69 0.62 63.65 0.01 0.87 
DRP 54.31 49.98 52.84 53.00 0.16 17.94 0.01 0.92 
CAHSEE 363.40 352.50 359.63 354.79 -4.84 42.06 -0.12 0.52 
Motiv. (Y4) 35.75 34.18 35.50 39.19 -0.31 7.61 -0.04 0.63 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted means 
by the standard deviation of the control group. CST-ELA=California Standards Test-English Language Arts; 
DRP=Degrees of Reading Power, CAHSEE=California High School Exit Exam. 


Table 43. Estimated Effect of Whole-School SLIC Intervention on Student Outcomes for 
Middle School Students (One Year Participation — Year 4) 


Raw Means Regression-Adjusted Diff Comparison SES _ p- 
Means Group SD value 
Comp. Interv. Comp. Interv. 
CST 358.98 356.80 356.94 358.61 1.67 55.49 0.03 041 
DRP 56.99 0.39 26,32 54.11 -2.21 14.31 0.15 0.31 
Motiv. (Y4) 36.06 34.10 0) 34.50 -1.20 7.91 -0.15 0.01 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted means 
by the standard deviation of the control group. 
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Table 35. Estimated Effect of Whole-School SLIC Intervention on Student Outcomes for 
High School Students (One Year Participation — Year 4) 


Raw Means Regression-Adjusted Diff Comparison SES _ p- 
Means Group SD value 
Comp. Interv. Comp. Interv. 
CST 348.53 346.80 348.29 352.45 4.16 57.85 0.07 0.58 
DRP ey eL 60.25 58.08 60.91 2.83 16.90 0.17 0.04 
Motiv. (Y4) 35.53 34.81 35.29 35.38 0.09 7.74 0.01 0.59 


Note: Standardized effect sizes (SES) are calculated by dividing the difference in regression-adjusted means 
by the standard deviation of the control group. CST-ELA=California Standards Test-English Language Arts; 
DRP=Degrees of Reading Power. 


Discussion 


Significant differences were found between high school students in the whole-school SLIC 
intervention group (intervention schools) compared to students in the control group (comparison 
schools) (p<.05). High school students in interventions schools had higher adjusted DRP scores 
(adjusted mean DRP: 60.91) when compared to the comparison high school students (adjusted 
mean DRP: 58.8). The adjusted CST-ELA scores for both intervention and comparison school 
students increased between their pre- and post-test administrations. However, the difference 
between the gains did not reach statistical significance. As a result, no significant differences 
between students in the whole-school SLIC intervention group (intervention schools) and 
students in the control group (comparison schools) were found for CST-ELA. For student 
motivation, significant differences were found between comparison school and intervention 
middle school students, with comparison middle school students having higher student 
motivation scores (35.7 vs. 34.5, p <.01). 
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APPENDIX A: IMPACT ANALYSIS METHODS 


Decision Rules for Inclusion/Exclusion of Covariates 


Inclusion of covariates in the study design may help to reduce possible selection bias in teacher 
and/or school selection. We identified pool of candidate variables (e.g., baseline pre-test scores — 
DRP, CST-ELA, CELDT, percent w/ BA as proxy for students socioeconomic status) based on 
educational research theories, research literature, and experience in working with the SDUSD. 
These variables were a priori selected to always be in the model (e.g., baseline test score). 


Following suggestions from Abt (Cris Price, From Data to Analysis: Common Issues across 
Sites, Striving Readers Local Evaluator conference, March 21-22, 2007), we used the following 
process: 


1. Identified pool of candidate variables (e.g., baseline pre-test scores - DRP, CST- 
ELA, CELDT, percent w/ BA as proxy for students socioeconomic status) based on 
educational research theories, research literature, and experience in working with the 
SDUSD. 

2. Specified how each covariate would be coded. 

3. Identified a priori a set of selected variables that would always be in the model (e.g., 
baseline test score). 

4. Performed sensitivity analysis to ensure that the magnitude and significance of the 
estimated treatment effect was not highly dependent upon the decisions to exclude or 
include certain covariates. 


Model Specification 


To assess the effect of the SLIC program on each of the student outcomes described above, six 
similar statistical models were estimated for the sample of students randomly assigned to 
treatment (in SLIC) or control (not in SLIC). If random assignment worked perfectly and all 
observations were independent from one another, the difference in the mean values for treatment 
and control would be the unbiased estimator of the effect of SLIC. Since the first assumption is 
not particularly strong, covariates are included to control for confounding variables that may be 
unbalanced due to imperfect random assignment. Since the assumption of independence is also 
not particularly strong in this case, a hierarchical linear model (HLM) is fit in order to allow for 
correlation of errors due to students being clustered by schools. 


For each outcome variable, the statistical model is 
y, =B,+Bl.x,+6,,+AK +6, (1) 


y 


where y,, is the outcome of interest for student i in school j, B, is the intercept, x,, is the 
treatment (1) or control (0) assignment of student i in school j, ¢,, 1s a random intercept 
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component, q@ is a vector of coefficients for a matrix of covariates K, and &; is the transitory 
error term as in a standard regression. 


For the CST-ELA, DRP, and student motivation models, the covariates included a measure of 
the dependent variable for the previous year (pre-test), a binary indicator variable for the 
student’s gender, an indicator of percent w/ BA as a proxy for socio-economic status, and a 
binary indicator variable for English learner status. In the analyses of standardized test scores, 
students were pooled by years of exposure to treatment or control rather than by school year. 
Since some students in the CST-ELA and DRP analyses experienced their first year of exposure 
to treatment or control in the first year of the study and others experienced their first year of 
exposure in the second year of the study, a binary indicator for students in the first year of the 
study is included in these analyses. Also, to account for differences between grade levels, grade 
level indicators for 8th, 9th, and 10th grade students (with 7th grade students as the comparison 
group) are included. 


For the CAHSEE, covariates were the same, except no pre-test was used. Since all students with 
available CAHSEE data received their first year of exposure in the second year of the study, and 
grade level was more constant among these samples, the indicator variables for grade level and 
cohort were not needed. Specific estimates for the covariates and random components are given 
in Appendix D. 


Tables Al and A2 show the response rates for selected outcome test measures for students in the 
targeted SLIC and control groups in two year participation and three year participation. 


Table Al. Response Rates for Targeted SLIC Students on Selected Outcomes (Two Year 


Participation) 

Variable Pre or Post Control Treatment Total 
CST-ELA Pre 99.55 99.23 99.40 
Post 88.91 86.07 87.48 

DRP Pre 80.18 84.42 82.33 
Post 70.54 70.10 69.58 

Motivation Y3 44.79 47.93 45.87 

Y4 45.02 44.63 44.82 


Note: SLIC= Strategies for Literacy Independence across the Curriculum, CST-ELA= California 
Standards Test-English Language Arts, DRP=Degrees of Reading, Motivation=Motivation Scale 
(12 items) in Year 3 and Year 4. 


Table A2. Response Rates for Targeted SLIC Students on Selected Outcomes (Three Year 
Participation) 
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Variable Pre or Post Control Treatment Total 
CST-ELA Pre 99.54 99.55 99.54 
Post 69.84 68.92 69.37 
DRP Pre 74.48 81.53 78.06 
Post 37.35 38.06 37.71 
Motivation Y3 47.80 50.68 49.26 
Y4 52.44 51.13 51.77 


Note: SLIC= Strategies for Literacy Independence across the Curriculum, CST-ELA= California 
Standards Test-English Language Arts, DRP=Degrees of Reading, Motivation=Motivation Scale 


(12 items) in Year 3 and Year 4. 


Table A3 shows the response rates for selected outcome test measures for students in the 


comparison schools compared to the intervention (SLIC) schools. 


Table A3. Response Rates for Selected Outcome Tests — Whole School Analysis (Y4) 


Variable Pre or Post Control Intervention Total 
CST-ELA Pre* 92.09 93.70 92.74 
Post 76.74 79.50 77.85 

DRP Pre* 74.40 80.75 76.96 
Post 48.68 55.54 51.45 

Motivation Y3 50.43 41.65 46.89 
Y4 53.71 46.56 50.83 


Note: CST-ELA= California Standards Test-English Language Arts, DRP=Degrees of Reading, 


Motivation=Motivation Scale (12 items) in Year 3 and Year 4. 


*Pre score is calculated one year prior to spring 2010. If spring 2009 pre score was missing spring 2008 


score was used instead. 


Note: 40.75% of the sample is intervention; 59.25% of the sample is control. 


131 


Tables A4, A5 and A6 show baseline characteristics for treatment and control groups for 
each set of analysis: Targeted SLIC with two year participation (A4), Targeted SLIC with 
three year participation (A5) and Whole School SLIC with one year participation in Year 4 


(A6). 


Table A4. Baseline Equivalence of the Treatment and Control Group across Key 


Characteristics (Two Year Participation - Targeted). 


Pre-CST-ELA 
Pre-DRP 
SES 
EL Status 
Female 
Grade: 

ath 

gth 

gi 


1 oe 


Control Treatment test-value* p-value 
298.0 296.7 0.96 0.3349 
39.5 A933 0.27 0.7873 
14.6% 15.1% -1.53 0.1251 
45.3% 48.3% 1.66 0.1981 
44.6% 44.0% 0.07 0.7976 
43.8% 42.1% 

11.9% 11.8% 

27.6% 29.3% 

16.8% 16.8% 0.77 0.8563 


* t-value for means and chi-square test for %. 


Note: Since parent education has a high rate of missing data (over 40%), SES is measured by 


percent with Bachelor’s degree according to student home zip codes. 
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Table A5. Baseline Equivalence of the Treatment and Control Group across Key Characteristics 


(Three Year Participation - Targeted). 


Pre-CST-ELA 
Pre-DRP 
SES 
EL Status 
Female 
Grade: 

ath 

gih 

gt 


1 oe 


Control Treatment test-value* p-value 
2992 299.8 -0.32 0.7518 
41.6 42.0 -0.43 0.6658 
13.9% 14.6% -1.52 0.1299 
43.4% 48.3% 2.06 0.1512 
45.2% 46.4% 0.12 0.7322 
27.6% 25.2% 
15.6% 15.3% 
35.5% 38.1% 
21.4% 21.4% 0.87 0.8327 


* t-value for means and chi-square test for %. 
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Table A6. Baseline Equivalence of the Treatment and Control Group across Key 
Characteristics (Whole School — Y4). 


Comparison Intervention test-value* p-value 
School School 
Pre-CST-ELA 345.7 343.3 -2.68 0.0074 
Pre-DRP 56.5 56.4 -0.26 0.7923 
SES 17.5% 16.4% -11.0 <.0001 
EL Status 24.9% 31.8% 90.0 <.0001 
Female 50.6% 46.1% 32.1 <.0001 
Grade: 
6" 11.9% 14.2% 
7 13.0% 15.7% 
gn 13.4% 14.9% 
gt 15.9% 14.7% 
10" 16.6% 15.1% 
11" 15.2% 14.1% 
12” 14.0% 11.3% 86.6 <.0001 


* t-value for means and chi-square test for %. 
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APPENDIX B: SURVEY ITEMS IN STUDENT MOTIVATION SCALE 


“T have the ability to complete my work and do well in school.” 


“T feel supported and respected by other students at my school.” 

“T have many opportunities to ask teachers questions about my work” 
“T place a high value on learning” 

“T put forth a great deal of effort when doing my school work.” 

“T think the things I learn at school are useful.” 

“T feel safe at school.” 

“My teachers believe I can do well in their classes.” 

10. “Overall, people at school accept me for who I am.” 

11. “If I were choosing a school again, I would still choose this one.” 

12. “I am satisfied with my classes.” 


$0 BO Oy OE 


“T care about rewards (grades, awards, etc.) that I get at school for my work.” 


Table B1 lists the component loadings from principal component analysis for the 12 motivation 


scale items. The 12 motivation items were combined into one composite score. 


Table B1. Principal Component Analysis for Motivation — Component Loadings 


Component One 


. [have the ability to complete my work and do well in school. 

. [care about the rewards (grades, awards, etc.) that I get at school for my work. 
. [feel supported and respected by other students at my school. 

. [have many opportunities to ask teachers questions about my work. 

. [place a high value on learning. 

. I put forth a great deal of effort when doing my school work. 

. I think the things I learn at school are useful. 

. I feel safe in school. 


Oo ANN FB WN 


. My teachers believe I can do well in their classes. 

10. Overall, people at school accept me for who I am. 

11. If I were choosing a school again, I would still choose this one. 
12. In general, I am satisfied with my classes. 


.618 
611 
.655 
659 
712 
.703 
659 
611 
.688 
615 
oF 
.683 
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APPENDIX C: SDUSD PERFORMANCE GOALS — TARGETED SLIC 
INTERVENTION 


The SDUSD identified several performance outcomes for the students participating in the 
targeted SLIC intervention. The evaluation team analyzed these performance outcomes cross- 
sectionally by calculating the percentage of SLIC intervention students who met these outcomes, 
by grade (as applicable) and for Year 1 (2006-07), Year 2 (2007-08), Year 3 (2008-09) and Year 
4 (2009-10). For purposes of determining enrollment, all students who received random 
assignments for a given year and who ended up in a targeted SLIC class are treated as part of 
intervention enrollment, regardless of their actual assignment. All students given random 
assignments and enrolled at the same school are treated as part of control enrollment. 


Goal 1. 100% middle school students make 2+ years growth in independent reading level 
per year enrolled (DRP). 


Actual performance Goal met? 
Yearl] Year2 Year3 Year 4* Year] Year2 Year3 Year 4* 
27.4% 22.5% 30.2% 55.6% No No No No 


Note: DRP=Degrees of Reading Power 
* In Year 4, no new random assignments were made. Hence, middle school only includes 8" graders with 
assignments continuing from 7” grade. 


Goal 2. 100% high school students make at least 2+ years growth in reading level per 
year enrolled (DRP). 


Actual performance Goal met? 
Year! Year2 Year3 £Year4 Year! Year2 Year3 Year4 
19.2% 27.7% 32.0% 50% No No No No 


Note: DRP=Degrees of Reading Power 


Goal 3. 100% of students will make yearly progress of at least one sublevel within a 
performance level until they reach “proficient.” 


Actual performance Goal met? 
Year! Year2 Year3 Year4 Yearl Year2 /Year3 Year4 


Grade 
T 54% 52% 48% n.a. No No No n.a. 
8 -- 43% 48% 56% No No No No 
9 61% 62% 57% 60% No No No No 
10 -- 31% 45% 41% No No No No 


Note: California Standards Test-English Language Arts (CST-ELA). The district has 
defined sublevels by dividing each performance level into equal thirds. Students already 
at “proficient” or higher on the pre-test are not included in this analysis. 
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Goal 4. 100% English learners will increase their oral proficiency level (OPL) by one 
proficiency level each year until reach advanced or are reclassified as Fluent English 
Proficient (FEP) (CELDT). 


Actual performance Goal met? 
Yearl Year2 Year3 Year4 Yearl Year2 Year3 Year4 
27.9% 31.5% 42.7% n.a. No No No n.a. 


Note: CELDT=California English Language Development Test. Post-test data are for the 
fall administration following the given study year. 


Goal 5. 100% will show positive attitudes toward reading, writing and perceptions of self 
as students. 


Actual Performance Goals Met? 


Survey Item Yearl Year Year Year4 Yearl Year2 /Year3 Year4 
2 3 

I like na. 32.7% 22.6% 24.6% naa. No No No 

reading. 

Ilike writing. na. 48.0% 45.1% 26.7% na. No No No 

I have the 

ability to 

complete my 

work and do 

well in No 

school. na. 62.0% 55.7% 514% na. No No 


Note: Data come from survey administered to students in all grades at study schools. 
Percentages represent responses in the top 2 categories of 5-point scales rating level of 
agreement with the statements in the table. 
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APPENDIX D: IMPACT OF TARGETED AND WHOLE-SCHOOL SLIC 
INTERVENTION ON STUDENTS 


(INDIVIDUAL REGRESSION ANALYSES) 


TARGETED SLIC - TWO YEAR PARTICIPATION 


Table D1. California Standards Test — Targeted SLIC, Two Year Participation 


Parameter 

CNT (n) 

TRT (n) 
Intercept 
Treatment 
Pre-Test 
Female 
English Learner 
Percent w/ BA 
Cohort Dummy Y1 
Cohort Dummy Y2 
8th Grade 


9th Grade 


10th Grade 


Full Sample 


(792) 
(782) 
133.79%** 
(12.33) 
0.40 
(1.72) 
0.57*## 
(0.03) 
3.44% 
(1.74) 
(1.90) 
0.45** 
(0.14) 
-0.05 
(6.02) 
1.27 
(1.96) 
5.65* 
(2.87) 
2.96 
(9.00) 
-10.38 
(9.09) 


English Learners 


(349) 
(358) 
121.41"? 
(14.84) 
1.55 
(2.50) 
O.58*** 
(0.05) 
0.98 
(2.55) 


0.86% 
(0.20) 
-0.29 
(6.93) 
0.23 
(2.93) 
4.38 
(4.39) 
1.36 
(8.55) 
-11.45 
(8.77) 


Middle Schools 


(426) 
(414) 
127.29** 
(14.4) 
1.96 
(2.18) 
0.59 
(0.04) 
3.47 
(2.21) 
-6.38** 
(2.42) 
0.47** 
(0.18) 
-0.45 
(5.61) 
-1.24 
(2.40) 
5.43% 
(2.65) 


High Schools 


(366) 
(368) 
145.37** 
(18.01 
-1.39 
(2.70) 
0.54% 
(0.04) 
3.37 
(2.75) 
-7,93** 
(2.99) 
0.44* 
(0.21) 


-13.10*** 
(2.87) 


Notes: Values are parameter estimates from hierarchical linear models, with standard 
errors in parentheses. ‘*’ denotes p < .05 and ‘**’ denotes p <.01 ‘***’ denotes p<.001. 
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Table D2. Degrees of Reading Power — Targeted SLIC, Two Year Participation 


Parameter 

CNT (n) 

TRT (n) 
Intercept 
Treatment 
Pre-test 
Female 
English Learner 
Percent w/ BA 
Cohort Dummy Y1 
Cohort Dummy Y2 
8th Grade 


Oth Grade 


10th Grade 


Full Sample English Learners Middle Schools High Schools 
(591) (266) (419) (172) 
(587) (280) (412) (175) 
24. 54* 22.30% 205184 18.46** 
(1.53) (1.76) (1.79) (3.14) 
0.90 22, 0.78 1.34 
(0.52) (0.73) (0.60) (0.95) 
O.s5*** O51 es Oa aa OS9th* 
(0.03) (0.04) (0.04) (0.06) 
-1.04* -1.61* -0.86 -1.59 
(0.51) (0.73) (0.60) (0.97) 
-2.20"** --- 2.44% -1.63 
(0.57) --- (0.66) (1.09) 
0.18*** V2255% 0.11* O.22*s* 
(0.04) (0.06) (0.05) (0.07) 
0.09 -0.08 -0.34 --- 
(1.14) (1.32) (1.14) --- 

1.09 209" 0.40 3.60** 
(0.60) (0.86) (0.67) (1.27) 
-1.70* -2.17 -1.42 --- 
(0.77) (1.15) (0.77) --- 

023 -0.37 --- --- 
(1.00) (1.20) --- --- 


Notes: Values are parameter estimates from hierarchical linear models, with standard errors in 


parentheses. ‘*’ denotes p < .05 and ‘**’ denotes p <.01 ‘***’ denotes p<.001. 
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Table D3. CAHSEE — Targeted SLIC, Two Year 
Participation 


Full English 
Parameter Sample Learners 
CNT (n) (242) (111) 
TRT (n) (248) (118) 
Intercept 362.3229" 331, 1677" 
(7.69) (7.44) 
Treatment -0.94 -0.20 
(2.78) (4.02) 
Female 4.80* tig 
(2.12) (3.27) 
English Learner -19.86*** --- 
(2.16) --- 
Percent w/ BA 0.39% 0.60* 
(0.16) (0.24) 


Notes: Values are parameter estimates from 
hierarchical linear models, with standard errors 
in parentheses. ‘*’ denotes p < .05 and ‘**’ 
denotes p < .01 ‘***’ denotes p < .001. 
CAHSEE=California High School Exit Exam. 
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Table D4. Student Motivation (Y4)- Targeted SLIC, Two Year Participation 


Parameter 

CNT (n) 

TRT (n) 
Intercept 
Treatment 
Motivation Y3 
Female 
English Learner 
Percent w/ BA 
8th Grade 


Oth Grade 


10th Grade 


Middle 

Full Sample —_ English Learners Schools 
(220) (100) (141) 
(231) (98) (146) 
LTS" e* 17.9404" 18.10** 
(1.89) (2.76) (2.45) 
0.17 1.05 0.47 
(0.88) (1.14) (1.29) 
O4aes* UAates OA ast 
(0.04) (0.06) (0.05) 
0.42 -0.25 0.85 
(0.76) (1.16) (0.98) 
0.31 --- -0.11 
(0.78) --- (1.01) 
0.04 0.02 -0.01 
(0.06) (0.09) (0.09) 
-0.19 -1.39 -0.15 
(1.25) (2.13) (1.29) 
1.11 1.17 --- 
(1.01) (1.82) --- 

OF72 1.30 --- 
(1.34) (2.49) --- 


High Schools 
(79) 
(85) 
18.64** 
(2.92) 
-0.73 
(1.28) 
0.44*** 
(0.07) 
-0.37 
(1.24) 
0.86 
(1.23) 
0.10 
(0.09) 


-0.03 
(1.33) 


Notes: Values are parameter estimates from hierarchical linear models, with standard 


errors in parentheses. ‘*’ denotes p < .05 and ‘**’ denotes p <.01 ‘***’ denotes p <0.001. 
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TARGETED SLIC - THREE YEAR PARTICIPATION 


Table D5. California Standards Test — Targeted SLIC, Three Year Participation 


Parameter Full Sample English Learners Middle Schools High Schools 
CNT (n) (301) (123) (173) (128) 
TRT (n) (305) (142) (171) (134) 

Intercept [33:82° 4% 166.60** 167.53** 176.00* 

(20.82) (26.03) (24.35) (32.75) 

Treatment -1.54 -2.30 0.81 -7.74 

(3.89) (4.72) (3.88) (10.26) 
Pre-Test O5124* 0.5474" 0.56*** 0.44% 
(0.06) (0.08) (0.07) (0.10) 
Female tila 293 8.01* 5.44 
(3.11) (4.76) (3.81) (5.17) 
English Learner -7.01* --- -8.09 -6.23 
(3.43) --- (4.28) (5.57) 
Percent w/ BA 0.25 0.65 0.34 0.11 
(0.24) (0.37) (0.31) (0.38) 
Cohort Dummy Y1 —-10.50 -14.08 -9.90 --- 
(8.08) (10.16) (7.46) --- 
8th Grade -27 87 *** -28.SO*** so OS Fe --- 
(4.40) (6.57) (4.06) --- 
9th Grade -33.94%** -35.02* --- --- 
(11.89) (13.97) --- --- 
10th Grade -83.38* --- --- -5.1.04 
(39.33) --- --- (41.07) 


Notes: Values are parameter estimates from hierarchical linear models, with standard 


errors in parentheses. ‘*’ denotes p < .05 and ‘**’ denotes p <.01 ‘***’ denotes p<.001. 
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Table D6. Degrees of Reading Power — Targeted SLIC, Three Year Participation 


Parameter 

CNT (n) 

TRT (n) 
Intercept 
Treatment 
Pre-test 
Female 
English Learner 
Percent w/ BA 
Cohort Dummy Y1 
8th Grade 


Oth Grade 


10th Grade 


Full Sample English Learners Middle Schools High Schools 
(156) (60) (155) (NA) 
(168) (88) (165) (NA) 
SD AS** ele Ree ad 34.34** --- 
(2.90) (3.73) (2.85) --- 
0.27 -0.04 -0.03 --- 
(1.02) (1.52) (1.00) --- 
Oarree 029% 0.39% --- 
(0.06) (0.08) (0.06) --- 
-0.81 -0.81 -0.76 --- 
(1.00) (1.47) (0.98) --- 
-3,.27** --- -3.17** --- 
(1.13) --- (1.11) --- 
0.23" OS a Ore --- 
(0.08) (0.12) (0.08) --- 
0.86 -0.02 0.95 --- 
(1.90) (2.32) (1.85) --- 
1.07 ibe 0:97 --- 
(1.12) (1.68) (1.09) --- 
-10.55* 1.91 --- --- 
(4.90) (9.26) --- --- 


Notes: Values are parameter estimates from hierarchical linear models, with standard errors in 


parentheses. ‘*’ denotes p < .05 and ‘**’ denotes p <.01 ‘***’ denotes p<.001. No cases for 


high school students. 
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Table D7. CAHSEE — Targeted SLIC, 
Three Year Participation 


Full English 
Parameter Sample Learners 
CNT (n) (93) (51) 
TRT (n) (99) (62) 
Intercept Bho Poe ie mS fo 8 a 
(7.48) (6.88) 
Treatment 0.14 2.83 
(4.80) (5.70) 
Female -0.06 -0.12 
(3.18) (4.32) 
English Learner -16.29*** -- 
(3.53) --- 
Percent w/ BA 0.34 0.32 
(0.24) (0.31) 


Notes: Values are parameter estimates from 
hierarchical linear models, with standard errors 
in parentheses. ‘*’ denotes p < .05 and ‘**’ 
denotes p < .01 ‘*** denotes p < .001. 
CAHSEE=California High School Exit Exam. 
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Table D8. Student Motivation (Y4)- Targeted SLIC, Three Year Participation 


Parameter 

CNT (n) 

TRT (n) 
Intercept 
Treatment 
Motivation Y3 
Female 
English Learner 
Percent w/ BA 
8th Grade 


Oth Grade 


10th Grade 


Middle 
Full Sample —_ English Learners Schools 
(125) (51) (64) 
(116) (44) (69) 
15.552°% 14724 15.35* 
(2.47) (3.75) (3.37) 
0.86 251 2.88 
(1.24) (1.79) (1.51) 
O52°%* 54% OSTts* 
(0.05) (0.09) (0.07) 
0.76 -0.65 0.53 
(1.02) (1.71) (153) 
-1.15 --- -2.79 
(1.04) --- (1.52) 
0.05 -0.01 -0.05 
(0.08) (0.12) (0.12) 
0.89 -3.06 0.77 
(1.54) (2.54) (1.60) 
-0.02 0.35 --- 
(1.41) (1.97) --- 
1.29 1.96 --- 
(1.70) (2.78) --- 


High Schools 
(61) 
(57) 
17.41* 
(3.42) 
-0.98 
(1.65) 
0.45*** 
(0.08) 
0.39 
(1.37) 
0.36 
(1.41) 
0.11 
(0.10) 


1.85 
(1.49) 


Notes: Values are parameter estimates from hierarchical linear models, with standard 


errors in parentheses. ‘*’ denotes p < .05 and ‘**’ denotes p <.01 ‘***’ denotes p <0.001. 
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WHOLE-SCHOOL SLIC INTERVENTION — ONE YEAR PARTICIPATION, YEAR 4 


(INDIVIDUAL REGRESSION ANALYSES) 


Table D9. California Standards Test - Whole-School SLIC, One Year Participation (Y4) 


Parameter 
CNT (n) 
TRT (n) 

Intercept 

Intervention School 

Pre-Test 

Female 

English Learner 

Percent w/ BA 

7th Grade 

8th Grade 

9th Grade 

10th Grade 


11th Grade 


12th Grade 


Full Sample 


(6823) 
(4915) 
47.118 
(3.96) 
2.78 
(3.72) 
0.85**# 
(0.01) 
3.12% 
(0.59) 
-2,00** 
(0.68) 
0.22% 
(0.05) 
3, 5g%* 
(1.05) 
7.08% 
(1.05) 
11.74%* 
(3.82) 
6.13 
(3.81) 
-1.80 
(3.82) 
-12.57 
(22.54) 


English Learners 


(2057) 
(1121) 
ALOT 
(5.07) 
0.62 
(3.86) 
0.87%#* 
(0.01) 
1.90 
(1.13) 


0.12 
(0.11) 

2.55 
(2.20) 

6.69** 
(2.13) 
12.25** 
(4.19) 
-5.30 
(4.16) 
-2.99 
(4.20) 
-6.83 
(31.93) 


Middle Schools 


(3003) 
(2427) 
AT .16*** 
(3.60) 
1.67 
(2.02) 
0.85% 
(0.01) 
2.13%* 
(0.82) 
(1.00) 
0.20** 
(0.07) 
3ST 
(1.00) 
7.06*** 
(1.00) 


High Schools 


(3820) 
(2488) 
58.44% 
(6.07) 
4.16 
(7.47) 
0.85% 
(0.01) 
3.94% 
(0.83) 
-1.05 
(0.93) 
0.26** 
(0.08) 


-17.90*#* 
(1.00) 
-13,55%## 
(1.03) 
-24.73 
(23.20) 


Notes: Values are parameter estimates from hierarchical linear models, with standard 
errors in parentheses. ‘*’ denotes p < .05 and ‘**’ denotes p <.01 ‘***’ denotes p<.001. 
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Table D10. Degrees of Reading Power — Whole-School SLIC, One Year Participation (Y4) 


Parameter 

CNT (n) 

TRT (n) 
Intercept 
Intervention School 
Pre-test 
Female 
English Learner 
Percent w/ BA 
7th Grade 
8th Grade 
9th Grade 


10th Grade 


11th Grade 


Full Sample English Learners Middle Schools High Schools 
(2859) (875) (990) (1869) 
(2234) (557) (828) (1406) 
4.00 3.43 Is:fa0" 6.66** 
(2.73) (3.84) (1.78) (1.28) 
0.34 0.15 -2.21 2.84% 
(1.42) (1.54) (2.17) (1.39) 
0.848% 0.88*** 0.844 0.848% 
(0.01) (0.02) (0.02) (0.01) 
0.43 0.41 0.03 0.66 
(0.26) (0.49) (0.38) (0.34) 
-0.91** --- > aks -0.46 
(0.30) --- (0.45) (0.39) 
O.1e# Oar 0.08* 0.155% 
(0.02) (0.05) (0.03) (0.03) 
aod ee --- -0.44 --- 
(5.98) --- (4.69) --- 
TAT 4.27 --- --- 
(2.77) (3.83) --- --- 

4.52 1.81 --- --- 
(2.39) (3.54) --- --- 
7.44% 4.96 --- 29 REOS 
(2.38) (3.53) --- (0.35) 
--- --- --- -4.43 
--- --- --- (2.53) 


Notes: Values are parameter estimates from hierarchical linear models, with standard errors in 
parentheses. ‘*’ denotes p < .05 and ‘**’ denotes p <.01 ‘***’ denotes p<.001. Not enough 


cases for 12th graders. 
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Table D11. CAHSEE -- — Whole-School SLIC, 


One Year Participation (Y4) 


Full English 
Parameter Sample Learners 
CNT (n) (1650) (638) 
TRT (n) (1005) (320) 
Intercept SoS DIT: 322.6088* 
(6.20) (7.37) 
Intervention 
School 4.69 -4.84 
(6.72) (7.59) 
Female Ae Fe ita 
(1.30) (2.32) 
English Learner =15.98"** --- 
(1.41) --- 
Percent w/ BA 0.8254" LO 
(0.13) (0.22) 
Grade 9 7.06 4.93 
(8.58) (13.29) 
Grade 10 26.64% ** 23.9300" 
(3.59) (4.51) 
Grade 11 -2.98 -1.48 
(4.11) (5.20) 


Notes: Values are parameter estimates from 


hierarchical linear models, with standard errors in 


parentheses. ‘*’ denotes p < .05 and ‘**’ denotes p < 
OL ‘***? denotes p < .001. CAHSEE=California High 


School Exit Exam. 
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Table D12. Student Motivation — Whole-School SLIC, One Year Participation (Y4) 


Parameter 
CNT (n) 
TRT (n) 

Intercept 


Intervention School 


Motivation Y3 


Female 


English Learner 


Percent w/ BA 


7th Grade 


8th Grade 


Oth Grade 


10th Grade 


11th Grade 


Middle 
Full Sample English Learners Schools 
(2906) (891) (924) 
(1662) (393) (768) 
IS g4es* LS.9088* Li250" 
(0.66) (1.12) (0.93) 
-0.40 -0.31 -1.20* 
(0.39) (0.64) (0.49) 
O.507t** O.567*** Oia4te* 
(0.01) (0.02) (0.02) 
O.OSek* 0.61 0.73* 
(0.19) (0.36) (0.33) 
025 --- 0.60 
(0.22) --- (0.39) 
-0.01 0.02 -0.02 
(0.02) (0.03) (0.03) 
-0.99* -0.64 -0.90** 
(0.48) (0.81) (0.34) 
-0.05 0.07 --- 
(0.48) (0.82) --- 
-1.08** -1.11 --- 
(0.38) (0.69) --- 
-0.56 -1.11 --- 
(0.34) (0.61) --- 
-0.17 0.04 --- 
(0.34) (0.60) --- 


High Schools 
(1982) 


(894) 
14.02*** 
(0.75) 
0.09 
(0.47) 
0.58% 
(0.01) 
1.04%* 
(0.24) 
0.05 
(0.26) 
0.01 
(0.02) 


-0.10 
(0.29) 
0.31 
(0.29) 


Notes: Values are parameter estimates from hierarchical linear models, with standard 


errors in parentheses. “*’ denotes p < .05 and ‘**’ denotes p <.01 ‘***’ denotes p <0.001. 


Not enough cases for 12th graders. 
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Table D13: National Curve Equivalents (NCEs) for adjusted California 


Standards Test (CST) 

Two Year Participation - Targeted Control Treatment 
Full Sample 33.8 34.0 
Middle School 33.4 34.3 

High School 34.6 34.0 
Three Year Participation - Targeted Control Treatment 
Full Sample Sh2 36.5 
Middle School 40.2 40.6 

High School 34.3 30.9 
Whole School — Y4 Comparison Intervention 
Full Sample 51.9 53.1 
Middle School 52.6 53.3 
High School 51.9 afore 


* NCE data not available for EL Students. 
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APPENDIX E: TARGETED IMPLEMENTATION FIDELITY 


Inputs 


Using reported and observed data on all of these forms of professional development, adequacy 
levels for professional development were determined for each intervention teacher and 
intervention coach. These levels are based on the following scales: 


Teacher PD Parti cipation 
1 = less than 120 hours/year [adequacy = low] 
2= 120-160 hours/year [adequacy = medium] 


3 = more than 160 hours/year [adequacy = high] 


Coach PD Participation 
1 = less than 80 hours/year [adequacy = low] 
2 = 80-150 hours/year [adequacy = medium] 


3 = more than 150 hours/year [adequacy = high] 


* Based on expectations stated in the Year | Implementation Report, full implementation would require 
approximately 200 hours of professional development for targeted intervention teachers. Cutpoints of 60 and 80% of 
this total are used here. For two teachers who split the academic year roughly in half, compliance levels are 
estimated based on reaching 50% of these cutpoints. 


™ Based on expectations stated in the Year 1 Implementation Report, full implementation would require 
approximately 190 hours of PD for coaches. Cutpoints of 40 and 80% of this total are used here. For one coach who 
was hired in November of Year 1, implementation is based on 100% of the yearlong expectations, as no other coach 
was at the site. 


™ Based on expectations stated in the Year 1 Implementation Report, full implementation would require 
approximately 190 hours of PD for coaches. Cutpoints of 40 and 80% of this total are used here. For one coach who 
was hired in November of Year 1, implementation is based on 100% of the yearlong expectations, as no other coach 
was at the site. 
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For the targeted intervention, a measure of support that site coaches provide to intervention 
teachers is based on estimates of the number of hours the coach spent with a teacher over the 
course of the year, typically in meetings between the coach and one or more intervention 
teachers to debrief lessons, examine student work, and plan upcoming lessons. Adequacy levels 
for coach-teacher support were determined for each targeted intervention teacher based on the 
scales below: 


Coach-to-Teacher Support 
1 = less than 216 hours/year [adequacy = low] 
2 = 216-288 hours/year [adequacy = medium] 


3 = more than 288 hours/year [adequacy = high] 


Calculation of Fidelity to the Classroom Model 


Fidelity to the program model of classroom instruction is based on the sum of five equally 
weighted components of instruction, each of which represents an important pedagogical 
expectation of the program. Site-level fidelity scores were determined on the basis of teacher- 
level fidelity scores, which were weighted by the number of intervention classes the given 
teacher taught A rough description of how each component was measured follows. 


Use of grade-level texts: A sample of texts used in instruction and gathered during the evaluation 
team’s observations was analyzed using Flesch-Kincaid Readability Test'. These scores were 
subtracted from the actual grade level of each class and the absolute differences were averaged 
across sites. Schools with average grade-text discrepancies of 2 grade levels or more were given 
a 1, those less than 2 but at least 1 were given a 2, and those with discrepancies less than 1 were 
given a 3. In general, use of texts above grade level was as much of a problem as use of texts 
below grade level. 


Scaffolding to independence: Teachers’ scaffolding students to independent literacy practice was 
assessed through classroom observation data on 3 aspects of instruction: the proportion of class 


“™ Based on expectations stated in the Year 1 Implementation Report, full implementation would require 
approximately 360 hours of direct teacher-coach support over the course of the year. Cutpoints of 60 and 80% of 
this total are used here. For two teachers who split the academic year roughly in half compliance levels are estimated 
based on reaching 50% of these cutpoints. 
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time the teacher spent on modeling literacy practices; the proportion of class time students had 
for independent practice; and the proportion of independent work time in which teachers 
circulated among and consulted with students individually. 


Coverage of curriculum: With respect to coverage of the SLIC curriculum, one point was 
awarded to each school because SLIC was the exclusive class curriculum. Teachers were given 
up to 1.5 points on the basis of material from interviews, surveys, and observations (up to .5 
points given for each source). An additional .5 points were given if coverage of the more 
“global” or abstract elements of the curriculum were taught thoroughly. In Year 3 data were 
combined using z-scores. 


Assessment of Needs/Differentiated Instruction: Assessment constitutes half of the calculation of 
assessment-differentiation, with another 1.5 points available for differentiation of instruction to 
respond to students’ needs. Based on the large amount of work spent on assessing student work 
in Year 2 and developing the assessment scoring system, each school was give full points for 
analysis of assessments and student work; these points were not automatically awarded in Year 


3, because cross-site assessment development decreased. In Year 2, the remaining 1.5 points 
were awarded for differentiation, and information on differentiation came from surveys, 
interviews, and observations. The 1.5 points were equally divided between SLIC classes at each 
school, and classes where differentiation was observed or reported were awarded that portion of 
the points. Schools with a SLIC/ EL class received an additional .2 to .22 points, and up to .3 
points were given for coaching support for differentiation. New schools and schools with new 
teachers and teachers new to SLIC had up to 0.2 points subtracted when the new teachers 
struggled with how to implement differentiation. 


Year 2 Metacognition: Metacognition is a diffuse characteristic of the type of instruction sought 
by the program. It is assessed here on a 10-point scale, based on data from classroom 
observations, scaled down to 3 points when pooled with the other components of instruction. 4 
points are the product of, on the one hand, the distribution of teacher's questioning across time 
and students and, on the other hand, the type of questioning, with the greatest emphasis on 
process questioning being given the highest rating. Two more points are based on individual 
ratings of "higher order" questioning and the teacher's probing of student responses. One point is 
based on the use of literacy journals in class. The remaining 3 points are based on a rating of how 


much the teacher encourages students to monitor their own progress. Site-level fidelity scores are 
based on the average of intervention-teacher scores across observed classes. 


Year 3 Metacognition: Metacognition is a diffuse characteristic of the type of instruction sought 
by the program. A score summing ratings for the distribution of questions across students and the 
distribution of questions across time (median number of questions in five minute blocks of 
whole-class instruction) is multiplied by a rating for the kind of questioning, with the highest 
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rating being given to classes where more than 50% of questions had a focus on literacy 
processes. A rating for the use of metacognition in independent work, when used (winter and 
spring observations), comprised an additional third of the rating. 


Site-level fidelity scores are based on the average of intervention-teacher scores across observed 
classes. The table below shows how fidelity scores would be calculated at a hypothetical site 
with 2 teachers, one of which taught 2 classes (Teacher 1), the other of which taught 3 (Teacher 
2). 
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Teacher-Level and Site-Level Fidelity Ratings by Inputs and Classroom Instruction (Targeted Intervention) 


i Coach-to-Teacher Support a 


Direct Indirect 


A. Teacher PD (“Coach-to- (“Coach PD B. Average Coach- | C. Classroom 


Participation* Teacher Support”) | Participation”) to-Teacher Support | Instruction Total Fidelity Score 


Rating Rating Rating Average Rating Rating 


School 1 


teacher I | Teacher 1 Score Teacher | Score Teacher | Score Teacher | Score A+B+C/3 
teacher 2 | Teacher 2 Score Teacher 2 Score Teacher 2 Score Teacher 2 Score A+B+C (icc 3 


Site score 
(Teacher 1 Score x 2) + See Column 1. Coach Score See Column 1. See Column 1. A+B+C/3 
(Teacher 2 x 3) 
5 


School 2, etc. 


teacher 1, 
ete. 
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Whole-School Implementation Fidelity Measures: Professional Development/Support 


The Year 1 Implementation Report suggests that content-area teachers at participating 
intervention schools should participate in a minimum of 1) 15-20 hours of “Whole-School/Cross- 
site conferences”; 2) 8 hours of “Small group, content-area seminars”; and 3) 8 hours of 
“Individual/in class support.” On average these expectations were met much less than 
expectations for implementation of the targeted intervention. 


In practice, it was not always possible to distinguish, for a given segment of professional 
development/support, which of the three categories of expectations for whole-school 
implementation was being met. Therefore, tables in the report present both category-specific 
estimates of total PD/support hours received by teachers at a given site and estimates of total 
hours of PD/support received. When sessions seemed to cross categories of expected PD/support, 
time was evenly divided among the categories. (For instance, a 4-hour session that provided both 
an overview of the intervention and discussion of content-specific literacy needs would count as 
2 hours of “whole-school conference” and 2 hours of ““content-area seminar.) 


Based on these guidelines, expected amounts of PD/support for content-area teachers were 
estimated as: 


more than 14 hours of “whole-school” PD, which included any PD with the primary purpose of 
presenting general goals or practices of the intervention; 


more than 7 hours of “content-area” PD, which included PD provided to teachers in one content 
area, focused on the individual literacy needs of that area; 


more than 7 hours of “individual/in-class support,” which included classroom observations, 
lesson-planning, and other individual-level support; and 


more than 27 hours of total PD/support. 
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APPENDIX F: PROGRAM EXPOSURE, YEARS 1 - 4 


Time Allotted for Intervention Classes 


SLIC is a supplemental class that students take in place of an elective class. There are exceptions 
to that, and two schools had control classes in Year 1, and three schools had control classes in 
Year 2. (Only one of the Year | schools kept its control class in Year 2). The time allotted to the 
intervention class varied by school, and the tables below display the average minutes per day that 
students could spend in SLIC classes in Years 1 and 2. 


Year | Intervention Classes: Average Minutes per Day 


School level N average SD minimum maximum 

middle school | 58.27 18.19 44.49 78.89 

high school 2 47.72 8.39 41.79 53.66 
total 5 54.05 14.71 41.79 78.89 


Year 2 Intervention Classes: Average Minutes per Day 


School level N average SD minimum maximum 
middle schools 4 56.34 18.19 42.60 78.50 
high schools 4 47.69 L238 41.39 54.26 
total 8 52.0 12.09 41.43 78.5 
Student Attendance 


Year 1 Attendance: Three middle schools participated in SLIC in Year 1. Averages for school 
attendance were taken from the School Accountability Report Cards (SARC) for 2006-07 
retrieved April 9, 2009 from http://studata.sandi.net/research/sarcs/index.asp There was no 
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SARC report for one of the middle schools in Year 1. Therefore, the average middle school 
attendance for Year | is based on the two schools for which we have data. Middle school 
attendance was calculated as (school | average annual attendance + school 2 average annual 
attendance)/2 and was 95.3%. The Year 1 high school average was calculated in the same way, 
and was 94.8%. 


Year 2, Year 3, and Year 4 Attendance: The SDUSD was able to provide the average annual 
attendance for each student enrolled in intervention and comparison schools for Year 2. The 
average attendance for each student is defined as the percent of days the student was present of 
the days they were enrolled at that school. In the tables below, the average middle school 
attendance is the average “percent present” for all middle school students at intervention schools 
rather than the average of school attendance averages. 


At both the middle and high school levels, the average attendance for SLIC students are slightly 
lower in most years than the average attendance for control students. The exception is Year 3 
middle schools. Attendance for those students who are ineligible for SLIC is higher still except 
in Year 4 middle schools. 


Average of Percent Present* - Targeted Intervention 


Ineligible 

School level SLIC Control Students Total 
Year 2 

middle school students 94.58 95.07 95.54 95.36 
high school students 91.6 93.15 93.49 93.25 
middle & high 93.18 94.11 94.39 94.22 
Year 3 

middle school students 94.9 94.8 95.2 95.0 
high school students 93.0 93.5 93.6 23,9 
middle & high 94.0 94.2 94.3 94.3 
Year 4 

middle school students 94.8 96.2 95.4 95.4 
high school students 93.0 93.6 94.1 93.8 
middle & high 93.3 94.1 94.8 94.5 


*Percent of days enrolled that student was present 
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Attendance data for Year 2 is additionally provided by grade level. Control students have 
average attendance that is equal to or slightly higher than SLIC student attendance, and the 
attendance for ineligible students is slightly higher than for either SLIC or control students in 
grades 7 and 9, but not grades 8 and 10. 


Year 2: Average of Percent Present* 


Grade SLIC 
10 93.8% 
11 
12 

Grand Total 93.20% 


Average Days Enrolled 


School level 
Year 2 


middle school students 
high school students 
middle & high 

Year 3 

middle school students 
high school students 
middle & high 

Year 4 

middle school students 
high school students 
middle & high 


Y2 assignment 


Control 


95.0% 
95.2% 
92.1% 
95.1% 


94.10% 
*Percent of days enrolled that student was present 


SLIC 


166.4 
160.7 
165.6 


173.0 
168.0 
170.6 


173.7 
167.8 
168.9 


Control 


168.4 
165.6 
166.9 


172.0 
169.9 
171.0 


176.8 
167.5 
169.4 


Ineligible 
Students 


NA 
95.7% 
95.0% 
92.3% 
93.0% 

NA 

NA 

94.30% 


Ineligible 
Students 


166.1 
160.9 
163.0 


172.7 
166.9 
169.5 


173.1 
168.1 
170.7 


Total 
NA 
95.4% 
94.9% 
91.9% 
93.4% 
NA 
NA 
94.20% 


Total 


166.4 
161.6 
163.8 


172.6 
167.3 
169.8 


1323 
167.9 
170.4 
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APPENDIX G: STUDY INSTRUMENTS 


San Diego Striving Readers Interview Protocols (teacher, coach) 
San Diego Striving Readers Classroom Observation Instrument (original) 
San Diego Striving Readers Classroom Observation Instrument (revised) 

San Diego Striving Readers Teacher Survey (Intervention Schools) 
San Diego Striving Readers Teacher Survey (Comparison Schools) 


San Diego Striving Readers Student Survey 
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Interview Protocol for SLIC Teachers 


May 21, 2010 


Introduction: 


The information you provide will become part of the study, but the report will not use the names of 
coaches or teachers. If there are any questions you don’t want to answer, just let me know and we’ll 
move on. 


BOLD Qs FOR NEW TEACHERS 
1. Tell me a little bit about yourself. What was your major in college? How long have you been a 


teacher? What topics do you teach? Grade levels? 


2. How did you hear about SLIC? How were you selected to be a SLIC teacher? Did you 
volunteer, or were you selected by your principal? 


3. How many Striving Readers/SLIC professional development hours have you participated this year 
(including during summer)? How many hours have you spent with your coach — how do you use the 
time? How many hours have you spent with developers/Rosemary — how do you use the time? 


4. Did you lead any PD this year? When was that, whom did you work with, how did that go? How did 
you prepare (on your own, with someone else? Who?) 


5. What do other teachers on campus think about SLIC/ how have they reacted to the program? 


6. How well do you feel you understand the SLIC intervention and your role in the intervention? 


7. Do you feel that you have received enough professional development to participate in the 
intervention? 


8. What additional type of professional development and/or information would be helpful? Are 
there other resources that would help you carry out the program? 
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9. About how many hours per week do you typically spend with your school’s literacy coach? How do 
you use the time? With the developer/ Rosemary? How do you use the time? Do you feel you’ve 
received enough support/guidance to implement the program? 


10. What do you see as the central ideas of SLIC? 


What literacy strategies do you think are unique to SLIC? 


12. What do you think are the strengths and weaknesses of the SLIC intervention? 


13. How well does SLIC work for EL students? Do you use SLIC the same way with EL students or do 
you use a different approach? Based on what you’ve seen, is the SLIC approach sufficient/enough to 
support (some) English Learners? Which aspects are especially helpful for them? If not, what other 
help do they need? 


14. Do you differentiate instruction for your students? What approach or approaches have you used, and 
how have they worked? How often do you do this? Can you give me an example of an instance in 
which you learned about a student’s literacy challenges, and describe how you differentiated for 
him/her? Does the coach/developer help either in planning or in carrying out differentiated 
instruction? Do you think you do this more/less/same as before SLIC? 


15. How do you go about teaching metacognition to students? How has that worked? Have you gotten 
help/suggestions/modeling from developers/coaches on how to go about that? Have you seen this 
“take” with students? Do they do it on their own, without prompting? 


162 


16. 


17. 


18. 


19. 


20. 


2 


— 


22. 


Has your instructional practice changed as a result of SLIC? How different is your 
teaching/instruction now as compared to before SLIC? Which parts of SLIC do you plan to keep in 
the future, and which would you drop? 


What have you covered so far this year in your SLIC class? For example, what topics, strategies, or 
instructional points, types of text, reading and writing strategies? 


Are there important things you haven’t covered? Or would like to cover? 


Does SLIC use grade-level texts? What determines whether a text is grade level? Do you use the 
texts provided in the curriculum units, or find others? 


How have the curriculum units worked this year — are they helpful? Which parts work best with your 
students, and which need further development/ do your students find hardest/ do you find easy/hard to 
teach? 


. How much do you use the curriculum units, how do you diverge from them? What do you cover 


that’s not in the units? 


Are your students independently using SLIC strategies in their content classes, so far as you know? 
(Why do you think that?) If not, what do you think are the barriers to the transfer of skills, strategies, 
knowledge? 
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23. Have you noticed any change in your students’ learning and/or behavior (including motivation, 
efficacy, engagement) as a result of your teaching the SLIC strategies? 


24. What do you think are the biggest challenges to implementing the key features of SLIC? 


25. How do you think these challenges can be overcome? 


[[[ 22. How do you think SLIC differs from the literacy strategies used in the Balanced Literacy 
Framework (SDUSD Blueprint). How does it differ from the use of genre blocks? ]] 


26. How much emphasis does the SLIC model place on the following aspects of literacy? Rate high, 
medium, low. 


- fluency 

- vocabulary 

- comprehension 

- phonics/ohonemic awareness/decoding 
- writing 


- student motivation 


27. Do you have any other comments you would like to share about the Striving Readers Initiative/SLIC 


at San Diego Unified School District? 
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Interview Protocol 


SLIC Coaches 5/18/10 


Introduction: 


The information you provide will become part of the study, but the report will not use the names of 
coaches or teachers. If there are any questions you don’t want to answer, just let me know and we’ll 
move on. 


SLIC & Change 


1. Has the curricular or pedagogical focus of SLIC shifted this year, and if so, how?— what ideas has the 
program focused on this year? 


Coach Support 


2. What is your role as a coach this year? If it changed, how and why did it change? From your 
perspective, how has it worked? -how much time/proportion of time do you spend on the targeted 
intervention, how much time on the whole school program 


3. How has leadership supported your work this year? How often do they visit, what do they provide, 
whom do they work with 


4. About how much PD have you had thus far (hours)? 
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Teachers 


5. How are your SLIC teachers doing? What do you see as their main strengths and needs? 


6. Have you noticed a change in teachers’ instructional practices — in SLIC or other classes? 


7. How do you work with your teacher(s) How much time do you spend with them, what issues do you 
deal with, and do you co-teach in SLIC classes? 


8. What have you covered so far this year in your SLIC classes? For example, what topics, strategies, or 
instructional points, types of text, reading and writing strategies? 


9. Are there important things you haven’t covered or would like to cover with your SLIC students? 


Students 


10. Have you noticed any change in your students’ learning and/or behavior (including motivation, 
efficacy, engagement) associated with SLIC? How have you observed that? How does the SLIC 
model/ practice address student engagement? 
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11. 


12. 


13. 


14. 


15. 


How well does SLIC work for EL students? Do you use SLIC the same way with EL students or do 
you use a different approach? Based on what you’ve seen, is the SLIC approach sufficient/enough to 
support (some) English Learners? Which aspects are especially helpful for them? If not, what other 
help do they need? 


Do you differentiate instruction for your students? What approach or approaches have you used, and 
how have they worked? How often do you do this? Can you give me an example of an instance in 
which you learned about a student’s literacy challenges, and describe how you differentiated for 
him/her? Does the developer/leadership help either in planning or in carrying out differentiated 
instruction? Do you think you do this more/less/same as before SLIC? 


How do you go about teaching metacognition to students? How has that worked? Have you gotten 
help/suggestions/modeling from developers/leader on how to go about that? Have you seen this 
“take” with students? Do they do it on their own, without prompting? 


Does SLIC use grade-level texts? What determines whether a text is grade level? Do you use the 
texts provided in the curriculum units, or find others? 


How have the curriculum units worked this year — are they helpful? Which parts work best with your 
students, and which need further development/ do your students find hardest/ do you find easy/hard to 
teach? 


Your School 
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[[ 1. What aspects of your school make it an easy or a challenging place to implement SLIC?]] How have 
people at your site responded to the program?]] 


16. How is the targeted intervention progressing at your school? How is the school-wide intervention 
going? (If there are challenges, how have you dealt with them?) 


17. What content PD have you offered at your school, and what plans do you have for PD at your school 
in the future? 


-what ideas do you think the content teachers have taken with them & implemented? 


- what do they believe the program to be/ be about? 


-are there differences in the way teachers respond to PD/offers of support — by content area, or by 
(e.g. grade level, gender)...? 


-what differences do you see in teachers’ implementation? 


Benchmark Assessment 


18. What is your view of the benchmarks & assessments — do they represent real skills, provide you with 
useful information about students &/or classroom instruction? 


19. How much time have you put into scoring the assessments this year? 
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20. How has the assessment/benchmark been used? e.g. does it contribute to differentiation (do you have 
an example?), or lesson planning (do you have an example?), or PD (can you give me an example?) 


General Issues 


21. What are the plans for next year — how might the program/your role change? 


22. How much emphasis does the SLIC model place on these aspects of literacy? Rate high, medium, or 
low 


- fluency 

- vocabulary 

- comprehension 

- phonics/ohonemic awareness/decoding 
- writing 


- student motivation 


23. Do you have any other comments you'd like to share about Striving Readers and SLIC? 


24. Do you have any questions or concerns about the study? 


Exit Questions 
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25. Has SLIC operated better/worse/ differently than other programs you’ve seen implemented? 


26. What ideas would you take with you (ideas about learning, about literacy); what do you see as the 
core ideas of SLIC? 


27. What classroom practices would you take with you/ leave behind? 


[[4. Based on your experience, how would you evaluate or assess SLIC]] 


28. How well have the social dynamics and organization of the program worked — onsite and among 
program personnel? 


29. Do you feel you were treated well, supported in your work? 


30. Did you have the resources you needed to succeed? 


31. Why did you decide to leave? 
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Striving Readers 
Principal Interview Protocol 


May 18, 2010 


Background 


With SLIC 


1. How did you become involved in SLIC? What did you know/what information were you given about 
the program? [how did you decide that your school would participate in SLIC? —if continuing 
principal] Did you know Rosemary, Trevor, or Chris before this? 


Personal Background 


1. Tell me a bit about your background — your education, your work as an educator and prior experience 
as an administrator. 


2. Do you have a background in literacy or particular ideas about adolescent literacy or EL? 


3. Have you worked with similar programs? If so, how is SLIC implementation similar to and different 
from those programs? 


SLIC 


1. If you were going to tell another professional about the SLIC program, how would you describe it? 
What are the essential ideas, processes, practices, and roles? 
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2. How do you see the principal’s role in the program -- What is your role in the program here? How do 
you go about encouraging content teachers to participate in the program? 


3. Whom do you communicate with in connection to SLIC (including district, coaches, etc)? What issues 
do you communicate about, and have you found that process productive/unproductive? 


4. What do you see as the strengths and weaknesses of SLIC? (any and all). 


Your School 


1. What other supplementary literacy programs does your school offer? 


2. What kinds of support programs/classes are provided for EL students? What other programs operate 
on campus? 


3. How have teachers, students, administrators at your site responded to the program? What is the 
reputation of SLIC on campus? 


4. What challenges are content teachers facing in understanding and implementing SLIC? Are the issues 
different across content areas? 


5. How is the targeted intervention progressing at your school? How is the school-wide intervention 
going? (If there are challenges, how have you dealt with them?) 


6. Have you found this school different in important ways terms of staff or student “culture” than other 
schools where you’ve worked — if so, how? (ask at Clairemont, small schools). 


Teachers 
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1. Have you noticed a change in teachers’ instructional practices or attention to literacy — in SLIC or 
other classes? 


Students 


1. Have you noticed any change in your students’ learning and/or behavior (including motivation, 
efficacy, engagement) associated with SLIC? How have you observed that? Have you had any 
interaction with parents about the SLIC program? 


2. How well does SLIC work for EL students? (ask at SIB?) 


3. How do students perceive the SLIC class -- do they see the class as different in some way? How have 
you observed that? Have you had interactions with parents about the program? 


General Issues 


1. Do you have any other comments you’d like to share about Striving Readers and SLIC? 


2. Do you have any questions or concerns about the study? 
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Interview Protocol 
District 


7/20/2010 


Background 


1. What are your and Trevor’s and Chris’s plans for Y5? 


2. What is the work in the Escondido district, how is it organized, how much of your time will it take, 
which parts of the SLIC model (literacy & implementation) are you taking with you and which are you 
leaving behind? 


3. What makes SLIC distinctive in relation to other adolescent literacy interventions? Which parts of 
SLIC are essential? 


4. Who is staying and who is leaving? And how will those school sites be organized next year? ? How 
has it worked without coaches (at Madison, Clairemont, Taft, ?CPMA?)? Could you have begun the 
program without coaches? 


Progress of SLIC 


1. How did the targeted intervention go — what worked well? What challenges did you experience in 
implementing the intervention, and how might you deal with those? Which aspects — of the model, of 
the implementation -- worked/did not work? 


2. How did the whole school intervention progress — what worked well and poorly? Did the 
presence/absence of a coach affect content area implementation? What challenges did you encounter? 
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3. Were there other accomplishments? 


4. How do you think that content area SLIC will be different next year? 


5. What plans are in place for sustainability? 


Leadership 


1. It seems that the PD/leadership/coach model has changed over the last 4 years. How would you 
describe the PD/coaching/leadership model in Y1 — Y4, and why did it change? 


2. What will the leadership model be in Y4? 


3. If you were to implement this program in the future, how would it be organized? 


4. Who works on the program’s interface with the principals — you, coaches, consultants?? How have 
those contacts worked? Where are they productive/ unproductive, and why? 


Y4 Program Elements 


1. How did these program elements work in Y4? coverage of curriculum, grade level texts, 
metacognition, assessment and differentiation, scaffolding to independence -- 


2. Was the academic community/student engagement idea implemented, did it work as you anticipated? 
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3. How did the curriculum books work, and the scoring guides? Do you plan to use them in Escondido? 


4. Were there MS/ HS differences? — general or local reasons? 


5. How much emphasis does the SLIC model place on these aspects of literacy? Rate high, medium, or 
low 


- fluency 

- vocabulary 

- comprehension 

- phonics/ohonemic awareness/decoding 
- writing 


- student motivation 


Teachers/teaching 


1. How did the SLIC teachers do in Y4 — what are their strengths and weaknesses? Which ones did 
particularly well? How did the new teachers perform — did they understand SLIC going in? If you 
were to rate each/all on a high/medium/low fidelity scale, how would you rate classroom fidelity in 
each class? 


2. What challenges are content teachers facing in understanding and implementing SLIC? Are the issues 
different across content areas? Or different between MS and HS? 


3. Have you noticed a change in teachers’ instructional practices — in SLIC or other classes? 


Coaches 
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1. How did the coaches do last year? — what are their strengths and weaknesses in understanding SLIC, 
and in carrying out a coaching role? 


2. What are your thoughts about what makes a good coach? A good SLIC teacher? 


3. What are the arrangements for coaches next year? 


4. Have interpersonal dynamics affected the implementation? Last year you mentioned some of the issues 
with interpersonal dynamics. (e.g. among coaches, etc.) How did that work this year? Have you had 
any further thoughts about how that went awry? 


Students 


1. How well does SLIC work for EL students? Do you use SLIC the same way with EL students or do 
you use a slightly different approach with them? 


2. Which students does SLIC work best with? From what you’ve seen do students who have made less 
progress in SLIC have literacy issues that are not addressed by SLIC, or do they need more SLIC, or 
have non-literacy problems? 


3. Have you noticed any change in students’ learning and/or behavior (including motivation, efficacy, 
engagement) associated with SLIC? How have you observed that? 


4. Would you give an example or anecdote of an instance in which you saw a student make substantial 
progress over the last year or years, and how various factors (including SLIC) played into the change? 
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Schools 


1. How would you assess the progress made at each SLIC school— what are the challenges and successes 
at each site? 


2. What aspects of these schools make them easy or challenging places to implement SLIC? 


3. Can this program be easily replicated elsewhere — if so, how given the amount of high level PD and 
intensive leadership involvement? 


General Issues 


1. Do you have any other comments you’d like to share? 


2. Do you have questions or concerns about the study? 
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Interview Protocol 
Consultants 


8/25/2010 


Background 


1. What are your plans for Y5? 


2. What is the work in the Escondido district, how is it organized, how much of your time will it take, 
which parts of the SLIC model (literacy & implementation) are you taking with you and which are you 
leaving behind? 


3. What makes SLIC distinctive in relation to other adolescent literacy interventions? Which parts of 
SLIC are essential? 


4. Who is staying and who is leaving? And how will those school sites be organized next year? Which 
coaches would you have liked to keep? How has it worked without coaches (at Madison, Clairemont, 
Taft, ?>CPMA?)? Could you have begun the program without coaches? 


Progress of SLIC 


1. How did the targeted intervention go — what worked well? What challenges did you experience in 
implementing the intervention, and how might you deal with those? Which aspects — of the model, of 
the implementation -- worked/did not work? 


2. How did the whole school intervention progress — what worked well and poorly? Did the 
presence/absence of a coach affect content area implementation? What challenges did you encounter? 


3. Were there other accomplishments? 


180 


4. How do you think that content area SLIC will be different next year? 


5. What plans are in place for sustainability? 


Leadership 


1. It seems that the PD/leadership/coach model has changed over the last 4 years. How would you 
describe the PD/coaching/leadership model in Y1 — Y4, and why did it change? 


2. What will the leadership model be in Y4? 


3. If you were to implement this program in the future, how would it be organized? 


4. Who works on the program’s interface with the principals — you, coaches, consultants?? How have 
those contacts worked? Where are they productive/ unproductive, and why? 


Y4 Program Elements 


1. How did these program elements work in Y4? coverage of curriculum, grade level texts, 
metacognition, assessment and differentiation, scaffolding to independence -- 


2. Was the academic community/student engagement idea implemented, did it work as you anticipated? 


3. How did the curriculum books work, and the scoring guides? Do you plan to use them in Escondido? 
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4. Were there MS/ HS differences? — general or local reasons? 


5. How much emphasis does the SLIC model place on these aspects of literacy? Rate high, medium, or 
low 


- fluency 

- vocabulary 

- comprehension 

- phonics/ohonemic awareness/decoding 
- writing 


- student motivation 


Teachers/teaching 


1. How did the SLIC teachers do in Y4 — what are their strengths and weaknesses? Which ones did 
particularly well? How did the new teachers perform — did they understand SLIC going in? If you 
were to rate each/all on a high/medium/low fidelity scale, how would you rate classroom fidelity in 
each class? 


2. What challenges are content teachers facing in understanding and implementing SLIC? Are the issues 
different across content areas? Or different between MS and HS? 


3. Have you noticed a change in teachers’ instructional practices — in SLIC or other classes? 


Coaches 
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1. How did the coaches do last year? — what are their strengths and weaknesses in understanding SLIC, 
and in carrying out a coaching role? 


2. What are your thoughts about what makes a good coach? A good SLIC teacher? 


3. What are the arrangements for coaches next year? 


4. Have interpersonal dynamics affected the implementation? Last year you mentioned some of the issues 
with interpersonal dynamics. (e.g. among coaches, etc.) How did that work this year? Have you had 
any further thoughts about how that went awry? 


Students 


1. How well does SLIC work for EL students? Do you use SLIC the same way with EL students or do 
you use a slightly different approach with them? 


2. Which students does SLIC work best with? From what you’ve seen do students who have made less 
progress in SLIC have literacy issues that are not addressed by SLIC, or do they need more SLIC, or 
have non-literacy problems? 


3. Have you noticed any change in students’ learning and/or behavior (including motivation, efficacy, 
engagement) associated with SLIC? How have you observed that? 


4. Would you give an example or anecdote of an instance in which you saw a student make substantial 
progress over the last year or years, and how various factors (including SLIC) played into the change? 


Schools 
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1. How would you assess the progress made at each SLIC school— what are the challenges and successes 
at each site? 


2. What aspects of these schools make them easy or challenging places to implement SLIC? 


3. Can this program be easily replicated elsewhere — if so, how given the amount of high level PD and 
intensive leadership involvement? 


General Issues 


1. Do you have any other comments you’d like to share? 


2. Do you have questions or concerns about the study? 
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SLIC CLASSROOM OBSERVATION INSTRUMENT (ORIGINAL) 
Classroom Observation Checklist 


Date: Grade Level: 7th 8th 9th 10th 11th 
School: Time/Period: Number of students: 
Adults present, their roles: 

Course//Subject Type 

OSLIC UO Control (Literacy Advancement Academy) O) Science O Math 

CO) History/social studies CO) English Language Arts Other 

Text sources used during class: 

1.0) Textbooks: O Literacy Science OJ Math CO History/social science O English 
Or: O1 Newspapers O Magazines [1 Novels UO Short Stories [Documents UO Online 


Other (e.g. essays, reviews, editorials, cartoons) 
2. Text Description re grade level 


O NONE 
Class Activity 
O Read Other Class 
Metivilies ___% of Class Time spent on literacy 
Text form: (e.g in content 
O Expository classes) SLIC topic 
O Persuasive 
Scena: econ (check all that apply, rate 1-3 for each) 
O Procedural a ae ___ Text features (headings, subheadings, captions, title) 


O Write 


Activity/task: 

O Notes 

O Summarize 

O Categorize 

O Analyze, synthesize, 
evaluate, etc. 

CO Review/Edit 

O Creating Charts 

O Reports 

ORespond to prompts 
O Essays, reviews, etc. 
OLiteracy journals 


O Letter/ application, etc. 


0 Lab reports 
O Formulas/ problems 


Based on: 

O One text 

OO Two texts 

O Three or more 


O Online work 
O Experiment 
O Inquiry 

O Demo 

O Discussion 


Other: 


Write the 
Main Topic or 
Focus of the 
Lesson: 


___ Text form (expository, narrative, persuasive, procedural) 

___ Text structure (pro-con, problem-solution) 

___ Paragraph structure (&/or topic sentence, supporting details) 
___ Main idea 

___ Cross-checking 

____ SLIC word functions (signal words/phrases; transitional 
phrases; language markers (for example, in summary, said, thought) 
Inference 

Analysis 

Synthesis 

Evaluation/ critique 

Vocabulary 

Din context 

Omorphology 

Oprior knowledge 

0 resources/texts 

Ogive definitions 


Other Literacy Topic 

____ Literary devices 
Figures of speech 
Authorial intent 
Rhetorical strategies 

___ Plot/character/setting 

____ Theme 
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Classroom Observation Ratings 


Overall Classroom Instruction [rate 1 — 5; where ‘5’ is highest/best] 


Classroom Instruction 

Building Literacy Knowledge. Are students taught literacy skills and strategies that support reading and writing in 
different text forms and in different academic disciplines? Are they taught in a way that emphasizes or furthers one 
of SLIC’s instructional purposes or goals (access to text, extraction of information/main idea, synthesis/analysis of 
text(s), critical thought — including evaluation, inference, and understanding of authorial intent)? If applicable, 
underline or circle one or more of the ‘literacy-related instructional purposes’ above. 


SLIC/other literacy classes 

(J Literacy skills and strategies are taught in a way that emphasizes one or more of the purposes listed above. 

(_] Literacy skills and strategies are taught in a somewhat routine manner, with little support for purpose or objective. 
J Literacy is not taught through a skills-strategies approach. 


Content-area classes 

(J Literacy skills and strategies specific to content-area texts and tasks are taught, and there is reading or writing practice. 
(J There is reading or writing practice, but no instruction in skills or strategies, or there is instruction in skills or strategies 
without reading or writing practice. 

(J There is no skills or strategies instruction, and there is no reading or writing practice. 


Building Independence. Are students supported as they practice reading and writing independently or in small 
groups? 


(J support is offered to students for a majority of time during small group work and/or independent practice. 
(J support is offered to students for a portion of time during small group and independent work 
(J little or no support is offered, or the teacher circulates but does not work with students. 


Achieving Independent Work. What percent of students appear to be on-task during independent work? 


(_] between two thirds and all of the students appear to be on task for most of the independent work period 
(-] between one and two thirds of the students appear to be on task for most of the independent work period 
_] 33% or less of the students appear to be on-task during most of the independent work period. 


Making Connections, & Making Instructional Points Explicit. Does the teacher make instructional points clear by 
stating them before or after lessons, putting them on wall charts, revisiting points from previous lessons, 
connecting instructional points to each other within and across classes, and/or drawing attention to the relevance of 
points to practice in other classes? 


(J major instructional points and connections are taught in a clear and thorough way 
(J instructional points and/or connections are taught, but lacking in clarity or thoroughness 
CJ no attention to instructional points 


Encouraging Self-Monitoring of Progress. Does the teacher encourage students’ monitoring of their own progress 
by talking about this progress or by giving them opportunities to assess their own progress? 


(J self-monitoring of progress is a noticeable feature of this lesson. 
(J a cursory or limited effort is made to encourage self-monitoring of progress. 
[_] there is no evidence that students are encouraged to monitor their own progress. 


186 


Academic Rigor. Is the teacher rigorous in the sense that he/she calls on students to think about their own thinking 
and to articulate that thinking using academic/technical terms; introduces challenging topics/tasks; asks 
challenging questions; and probes student comments and responses (e.g., through follow-up questions). 


(J high academic rigor 
_] medium/typical academic rigor 
(] low academic rigor 


Distribution of Questioning— if it occurs. 

(J Questioning involves (or is actively directed) at most students. [directed at = teacher asks named students] 
(J Questioning involves (or is actively directed) at least a third of the students. 

(_] Questioning is confined to just a few students. 


Kind of Questioning (literacy-specific) 

(J Process questioning is a major focus of instruction (at least half of questions are process questions*). 
_] Process questioning is a minor focus of instruction (>1/5, < 1/2 of questions are process questions). 
(J There is very limited use of process questioning. 

(J There is no process questioning. 


* Calling for a response about literacy process or framed in terms of specific processes (e.g., [following a preview of text 
features] “What do you think this article will be about?”). Content questions, by contrast, only call for understanding of the text 
content and could, at least in some cases, be answered on the basis of prior knowledge, recall, or one’s own opinions. This may 
be a difficult distinction to make on a question-by-question basis. When it is, the issue should be whether the teacher is trying to 
get students to use and think about particular literacy practices or is just asking students to give answers without respect to the 
(literacy-related) method of arriving at the answer. In the latter case, of course, questions should not be seen as process 
questions. 


General Teaching Skills (check mark or “X’’) 
high medium low 


teacher clarity 
teacher enthusiasm 
task-oriented behavior 
teacher varies lesson approaches 
teacher prepares/informs re testing 
teacher asks higher order questions 
*teacher uses student ideas 
teacher probes student comments 
*lesson organization/structure 
* Part of the SLIC pedagogy, in addition to being identified as a teaching skill associated with student progress 
(citation). 


Classroom Atmosphere/Behavior (check mark or “X”) 
high medium low 


*established routines 
**respectful behavior 
*part of SLIC pedagogy 
** not mentioned by SLIC or other sources 


Planning Sessions (Include in notes: student work/diagnostics/ classroom performance is discussed; 
discussion of individual and common learning needs and areas of progress; planning to address learning needs.) 


Is a planning session held? [] Yes [No 
Was the observer able to attend the planning session? [] Yes _[_] No 
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Classroom Observation Timed Sheet 


For “topic”: L=literacy, V = other instructional topics, Blank=not instructional 
For “whole class”: T= apes predominately S = student(s) predominately 
For “circulating/supporting” : T = teacher C = coach O=other 


Small Inde- Circulate | Questio Notes 
Model/ groups pendent ns 
Lecture Chart T/ C/O | Work 


5-10 


10-15 


15-20 


20-25 


25-30 
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CLASSROOM OBSERVATION INSTRUMENT (REVISED) 
Classroom Observation Checklist 


Observer: 


School: Teacher: 


Date: 


Grade level: Time period: Number of students: 


Other adults present, their roles: 


Class type: 


Text type: 


CO Newspaper 0) Magazine O Novel OU Short story 


Text info (including description, title, topic, 
grade level, etc.): 


OO Textbook - type: O) Other - type: 

OVERVIEW LITERACY TOPICS 
Text Form Build/use explicit literacy knowledge through... 
Reading 


0 Expository 
CO Persuasive 
O Narrative 
O Other 
describe: 


Writing 

CO Expository 
CO Persuasive 
O Narrative 
O Other 
describe: 


Test-prep: 
0 Type: 


describe: 


Summary: 
Total min. 
of class: 


Total min. 
of non-lit.: 


Write Lesson 
Descrip. on 
reverse side 
of this sheet. 


Reading 
O anticipate content while reading 
CO locate content 
CO preview text to build an understanding of content prior to reading 
CO) cross-check 
CO identify and/or analyze authorial intent 
CO integrate information from text features with running text 
CO locate and/or develop understanding of main ideas 
CO analyze text 
O evaluate/critique text 
Writing 

O) gather and organize information (note-taking) 

0 using text features 0 using research questions 

OD other: 

O from single sources O) from multiple sources 
O make explicit decisions about text structure 
0 use text features in writing as tools for conceptualizing and thinking critically 
CO) develop coherent paragraphs 
0 develop relevant research questions 
O) develop controlling thesis 
0 use writing (beyond note-taking) to... 

O organize information 0) analyze information 

O evaluate/critique information O synthesize information 

O from single sources O) from multiple sources 
O) support claims or arguments with evidence 
0 anticipate & address readers’ concerns, counterclaims, misunderstandings or biases 


Build knowledge of language conventions and vocabulary knowledge by... 
O making meaning in unfamiliar vocabulary 
O context O morphology O grammar 
O other: 
0 using vocabulary appropriate to purpose, audience, and content 
O revising drafts for clarity, accuracy in spelling, punctuation, grammar, citations, etc. 


Miscellaneous 
O written reflection on literacy 
0 other topic: 
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Classroom Observation Ratings 


Overall Classroom Rating (rate 1 — 5, where 5 is highest/best) 


Building Literacy Knowledge 

O Instruction and/or practice in specific literacy skills and strategies is clearly driven by the purpose for reading or 
writing a piece (or pieces) of text. 

O Instruc and/or practice in specific literacy skills and strategies is provided that is relevant to the purpose for 
reading or writing, but the connection between that purpose and the skills/strategies being taught/practiced is 
somewhat limited. 

O Instruction and/or practice in specific literacy skills and strategies is not purpose-driven or the instruction and/or 
practice is, in and of itself, highly limited. 

O Instruction and/or practice in literacy skills is not occurring. 


Literacy Instruction 

O Instruction in literacy skills and strategies is a significant part of the lesson. 

O Instruction in literacy skills and strategies is a minor part of the lesson. 

O Instruction in literacy skills and strategies is either negligible or not a part of the lesson. 


Literacy Practice in Content-Area Classes 

O Reading or writing practice is a significant part of the lesson. 
O Reading or writing practice is a minor part of the lesson. 

O There is little or no reading or writing practice. 


Achieving Independent Work 

0 Less than 20% of behavior was off-task. 
O There was 20-50% off-task behavior. 

O) More than 50% of behavior was off-task. 


Classroom Environment 

O Behavior and/or classroom management does not impede whole group instruction and activities. 

O Behavior and/or classroom management somewhat impedes whole group instruction and activities. 
O Behavior and/or classroom management significantly impedes whole group instruction and activities. 


Encouraging Self-Monitoring 

O Self-monitoring is a noticeable feature of this lesson. 

O A cursory or limited effort is made to encourage self-monitoring. 
O There is no evidence that students are encouraged to self-monitor. 


Metacognitive Activity 

O Independent activities provide explicit and substantial opportunity for students to be metacognitive about their 
knowledge and practice. 

O Independent activities provide explicit but brief opportunities for students to be metacognitive about their 
knowledge and practice. 

O Metacognition about knowledge and practices is not an explicit part of independent activities. 


Academic Rigor 
O High academic rigor. O Medium academic rigor. O Low academic rigor. 


Distribution of Questioning 

O The teacher involves 2/3 or more of the students in questioning. 
O The teacher involves 1/3 to 2/3 of the students in questioning. 
O The teacher involves less than 1/3 of students in questioning. 


Kind of Questioning 

O A majority of questions asked by the teacher are process questions (50% or more). 
O A minority of questions asked by the teacher are process questions (20% to 50%). 
O A small portion of questions are process questions (less than 20%). 
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Classroom Observation Timed Sheet 


0 
(1:00-1:05 | | 
(1:05-1:10 | | 
(1:10-1:15 | | 
(1:15-1:20 | | 
(1:20-1:25 | | 
(1:25-1:30 | | 
(1:30-1:35 | | 
(1:35-1:40 | | 
(1:40-1:45 | | 
(1:45-1:50 | | 
(1:50-1:55 | | 


SUMMARY INFO: 


P/Total ratio = 


19] 


